Evaluating the Performance of Static Code Analysers: The Gaps between Academia and Industry

Static Code Analysis (SCA) tools play a vital role in software validation and verification. However, no recent secondary study summarises existing primary studies that evaluate the performance of existing SCA tools. Such a study is vital to identify and inform relevant and useful SCA tools for the community. To address the gap, systematic mapping and grey literature studies were conducted, comparing existing SCA tools evaluated in academia and by the software engineering practitioner community to generate insights into their performance. We performed detailed meta-analyses of the outcomes from these studies, identifying the SCA tools studied, the performance metrics used to study them, the research approaches used to evaluate SCA tools in academia, and the best-performing SCA tools. We found that only 26.7% of the SCA tools reported in the practitioner community have been studied in academia, and there are significant gaps in academia evaluating SCA tools compared with the software engineering practitioner community. Additionally, the best tools reported in the software engineering community have high false-positive and high false-negative rates. Outcomes of our study also show divergent approaches among the academic and practitioner communities for evaluating the performance of SCA tools. With enhancement efforts, SCA tools may enhance developers’ workflows.