CryptoAPI-Bench: A Comprehensive Benchmark on Java Cryptographic API Misuses

Several studies showed that misuses of cryptographic APIs are common in real-world code (e.g., Apache projects and Android apps). There exist several open-sourced and commercial security tools that automatically screen Java programs to detect misuses. In order to compare their accuracy and security guarantees, we develop a comprehensive benchmark named CryptoAPI-Bench. CryptoAPI-Bench consists of 171 unit test cases that cover basic cases, as well as complex cases, including interprocedural, field sensitive, multiple class test cases, and path sensitive data flow of misuse cases. The benchmark also includes correct cases for testing false positive rates. We evaluate CryptoAPI-Bench on four tools, namely, SpotBugs, CryptoGuard, CrySL, and Coverity and present their performance and comparative analysis. Our benchmark is useful for advancing state-of-the-art solutions in the space of misuse detection.