Since the inception of Ethereum, numerous static analysis tools for Solidity-based smart contracts have been developed. During this time, Ethereum has rapidly evolved, undergoing 19 hard forks and five major breaking changes in the Solidity language, which can directly impact the capabilities of these tools. Existing research is often constrained to specific versions and tools, without offering a systematic and comprehensive evaluation of how these evolutions affect static analysis tools. However, assessing the full impact of these upgrades on the tools is far from straightforward. To address this, we present a large-scale comprehensive empirical evaluation that employs 22 state-of-the-art static analysis tools and 251,340 unique contracts verified on Etherscan to systematically evaluate the compatibility, adaptability and consistency of these tools throughout the evolution of smart contracts. First, we conduct a pilot study on a representative sample subset of the full dataset to evaluate the compatibility of the tools by examining their execution success rates across different versions. Our findings show that only 10 out of 22 tools are fully compatible with all versions. Based on these findings, we perform a large-scale evaluation on the full dataset to explore the adaptability of these 10 tools across different versions. Our results reveal that, out of 10 tools, nine do not maintain stable detection rates, five exhibit significant TP rate fluctuations, and seven show notable degradation in effectiveness. Finally, we further evaluate the consistency of these 10 tools by comparing their detection results on the full dataset, showing that consistency tends to decrease as the versions are upgraded. Additionally, we analyze the reason behind limitations of static tools and discuss the contract semantic drift over time, offering valuable insights for future research.
Gao et al. (Tue,) studied this question.