Priority Sampling of Large Language Models for Compilers

Large Language Models show great potential in generating and optimizing code. Widely used sampling methods such as Nucleus Sampling increase the diversity of generation but often produce repeated samples for low temperatures and incoherent samples for high temperatures. We present Priority Sampling, a simple and deterministic sampling technique that produces unique samples ordered by the model's confidence. Additionally, Priority Sampling supports a controllable and structured exploration process using regular-expression-based generation. Priority Sampling outperforms Nucleus Sampling for any number of samples, boosting the performance of the original model from 2.87% to 5% improvement over -Oz. Moreover, it outperforms the autotuner used to generate training labels of the original model in just 30 samples.