A Preliminary Evaluation of Text-based and Dependency-based Techniques for Determining the Origins of Bugs

A crucial step in understanding the life cycle of software bugs is identifying their origin. Unfortunately this information is not usually recorded and recovering it at a later date is challenging. Recently two approaches have been developed that attempt to solve this problem: the text approach and the dependency approach. However only limited evaluation has been carried out on their effectiveness so far, partially due to the lack of data sets linking bugs to their introduction. Producing such data sets is both time-consuming and challenging due to the subjective nature of the problem. To improve this, the origins of 166 bugs in two open-source projects were manually identified. These were then compared to a simulation of the approaches. The results show that both approaches were partially successful across a variety of different types of bugs. They achieved a precision of 29%-79% and a recall of 40%-70%, and could perform better when combined. However there remain a number of challenges to overcome in future development - large commits, unrelated changes and large numbers of versions between the origin and the fix all reduce their effectiveness.