On Designing Programming Error Messages for Novices: Readability and its Constituent Factors

Programming error messages play an important role in learning to program. The cycle of program input and error message response completes a loop between the programmer and the compiler/interpreter and is a fundamental interaction between human and computer. However, error messages are notoriously problematic, especially for novices. Despite numerous guidelines citing the importance of message readability, there is little empirical research dedicated to understanding and assessing it. We report three related experiments investigating factors that infuence programming error message readability. In the frst two experiments we identify possible factors, and in the third we ask novice programmers to rate messages using scales derived from these factors. We fnd evidence that several key factors signifcantly afect message readability: message length, jargon use, sentence structure, and vocabulary. This provides novel empirical support for previously untested longstanding guidelines on message design, and informs future eforts to create readability metrics for programming error messages.