Automated Processing of Referring Acts in Pair-Programming Dialogue

Pair programming is a collaborative technique where two people work together (in person or remotely) on the same piece of code. Numerous studies have demonstrated its benefits in industrial and educational settings, though they also highlight the challenges for its implementation. AI tools have been suggested as a way to overcome some of these obstacles, for instance by having a conversational agent act as a partner where no suitable human peer is available. With the advent of Large Language Models (LLMs), developing sophisticated conversational agents has become a simpler task. However, pair-programming dialogue presents some unique challenges, particularly due to its being a form of spoken dialogue and containing programming domain terminology. In this work, we looked at how LLMs understand pair-programming dialogue by looking at reference processing, as reference processing tasks represent a key structure in discourse, the attentional structure. We carried out experiments in mention detection, coreference resolution, and reference generation. In order to carry out our experiments, we first collected a dataset of remote pair-programming dialogues between students completing simple Python tasks. We collected speech data, keylog records, code files, and screen captures. We then transcribed the dialogues and combined the different data sources. Next, we developed an annotation scheme to annotate referring acts in the dialogues and trained a team of seven annotators on how to annotate the different layers of our scheme. Our experiments showed that recent LLMs could process references comparably to humans to some extent, with the characteristics of pair-programming dialogue sometimes hindering performance. We also observed that different LLMs can present completely different attentional structures, some more human-like and more suitable for the collaborative talk required in pair programming. With this work, we release a public dataset of pair-programming dialogues with rich contextual information and reference annotations, as well as the annotation scheme used to develop our dataset. We also contribute some empirical evidence about how LLMs process references in a challenging context, and which discourse features need to be taken into account when dealing with complex text genres. We also illuminate some links with how LLMs process references and their suitability for educational pair programming.