Unmasking the Type Inference Capabilities of LLMs for Java Code Snippets

Type inference is a crucial task for reusing online code snippets. Although code snippets are prevalently shared on platforms like StackOverflow, they frequently lack essential type information, such as fully qualified names (FQNs) and required libraries. Recent studies have leveraged Large Language Models (LLMs) to perform type inference for such code snippets, demonstrating promising performance. However, these evaluations may suffer from data leakage, as the benchmark suite, StatType-SO, used for evaluation has been publicly available on GitHub since 2017. Consequently, it remains uncertain whether the strong performance of LLMs reflects genuine semantic understanding of code or is due to the ground truth being included in the training set. This paper strives to comprehensively evaluate the genuine type inference capabilities of LLMs on Java code snippets and identify potential limitations of LLMs. First, we created ThaliaType, a new, previously unreleased benchmark suite designed for type inference evaluation. Second, using the StarCoder2 LLM as a baseline, we uncovered data leakage from StatType-SO in StarCoder2’s open-source training set and observed that other state-of-the-art LLMs exhibit similar performance drops when evaluated on ThaliaType, with precision decreasing by up to 59% and recall by up to 72%. Finally, we developed semantic-preserving code transformations to further investigate the capabilities of LLMs in understanding the execution semantics of code snippets. Our results showed that the performance of LLMs on StatType-SO is far less robust to these transformations than on ThaliaType, suggesting that the performance on StatType-SO may be biased by data leakage and have limited generalizability. These findings highlight the importance of carefully designed, leakage-free benchmarks for evaluating LLMs on type inference tasks. We recommend future studies adopt ThaliaType to ensure rigorous and reliable assessments of LLMs’ genuine type inference capabilities.