Programming without a Programming Language: Challenges and Opportunities for Designing Developer Tools for Prompt Programming

Existing tools for writing prompts for language models (known as “prompt programming”) provide little support to prompt programmers. Consequently, as prompts become more complex with the addition of multiple input/output examples (“few-shot” prompts), they can be hard to read, understand, and edit. In this work, we observe that prompts are often used to solve complex problems, but lack the strict grammar of a traditional programming language. We describe methods for extracting the semantically meaningful structure of natural language prompts (e.g., regions of the prompt representing a preamble or input/output examples) in the absence of a rigid formal grammar, and demonstrate a range of editor features that can leverage this information to assist prompt programmers. Finally, we relate initial feedback from design probe explorations with a set of domain experts and provide insights to help guide the development of future prompt editors.