CounterFault: Value-Based Fault Localization by Modeling and Predicting Counterfactual Outcomes

This paper presents a new, flexible approach to automatically localizing faults in software, named CounterFault. It uses a form of causal inference called counterfactual prediction to predict the effect, on the success or failure of an execution Ex, of intervening at a statement s to set an assignment target A to a value a that is not actually assigned to A in Ex but that could be if s or Ex was modified. CounterFault generates this prediction without actually modifying s or Ex, by employing a very flexible non-parametric statistical or machine learning model (e.g., a random forest). CounterFault applies this basic idea to estimate, with minimal confounding bias, the average causal effects on program failures of different changes in the values assigned to program variables, and these estimates are then employed to derive suspiciousness scores, which are used to assist developers in localizing faults. This paper also reports on an empirical evaluation of CounterFault involving the widely used Defects4J evaluation framework, which contains real software faults, as well as several other Java numerical programs. CounterFault is compared empirically with two other value-based fault localization techniques and four of the best performing coverage-based techniques. The results indicate that CounterFault is more effective than the competing techniques.