A Large Language Model Pipeline to Automate the Solution of Competitive Programming Problems

Abstract The recent rapid introduction of large language models has enabled new black box approaches to optimize the application of these models for various scenarios. GPT-4 is a multimodal large language model introduced by OpenAI which can answer complex questions, analyze nuanced data, and solve complicated programming problems. The performance of GPT is dependent upon the provided prompt and hyperparameters. This paper explores the effect of minor variations in system prompt and parameters including temperature and top-p for code generation and code accuracy for competitive programming tasks. Temperature controls the amount of randomness in the response, with a temperature of zero producing deterministic output. Top-p controls the cumulative probability distribution for tokens considered for the next output token. Based on the results, we propose approaches to optimize system prompts for code-generation and parameter values to improve the correctness of code. In addition, we propose a pipeline that utilizes these enhancements to effectively solve algorithmic puzzles common in computer science education, in addition to complex contest programming problems.