What question did this study set out to answer?

This research aims to assess the performance of various large language models in assisting with U.S. tax resolutions, particularly for low-income taxpayers.

February 9, 2026Open Access

Performance of LLMs on VITA test: potential for AI-assisted tax returns for low income taxpayers

Key Points

This research aims to assess the performance of various large language models in assisting with U.S. tax resolutions, particularly for low-income taxpayers.
Compared diverse large language models using VITA certification tests endorsed by the IRS.
Developed ground truths for 130 questions with tax domain experts.
Benchmark models against original and syntactically perturbed VITA questions.
Evaluated model performance on both basic multiple-choice and open-ended questions.
Closed-source models generally showed higher accuracy than open-source models.
Top models like GPT-4 and Claude 3.5 Haiku achieved 83.33% accuracy on multiple-choice questions.
GPT-3.5 and Llama 3.1 8B had lower accuracy, performing at 54.17% and 50.00%, respectively.
All models struggled significantly with open-ended questions requiring numerical calculations.

Abstract

Abstract This paper investigates the performance of a diverse set of large language models (LLMs) including leading closed-source (GPT-4, GPT-4o mini, Claude 3.5 Haiku) and open-source (Llama 3.1 70B, Llama 3.1 8B) models, alongside the earlier GPT-3.5 within the context of U.S. tax resolutions. AI-driven solutions like these have made substantial inroads into legal-critical systems with significant socio-economic implications. However, their accuracy and reliability have not been assessed in some legal domains, such as tax. Using the Volunteer Income Tax Assistance (VITA) certification tests—endorsed by the US Internal Revenue Service (IRS) for tax volunteering—this study compares these LLMs to evaluate their potential utility in assisting both tax volunteers as well as taxpayers, particularly those with low and moderate income. Since the answers to these questions are not publicly available, we first analyze 130 questions with the tax domain experts and develop the ground truths for each question. We then benchmarked these diverse LLMs against the ground truths using both the original VITA questions and syntactically perturbed versions (a total of 390 questions) to assess genuine understanding versus memorization/hallucinations. Our comparative analysis reveals distinct performance differences: closed-source models (GPT-4, Claude 3.5 Haiku, GPT-4o mini) generally demonstrated higher accuracy and robustness compared to GPT-3.5 and the open-source Llama models. For instance, on basic multiple-choice questions, top models like GPT-4 and Claude 3.5 Haiku achieved 83.33% accuracy, surpassing GPT-3.5 (54.17%) and the open-source Llama 3.1 8B (50.00%). These findings generally hold across both original and perturbed questions. However, the paper acknowledges that these developments are initial indicators, and further research is necessary to fully understand the implications of deploying LLMs in this domain. A critical limitation observed across all evaluated models was significant difficulty with open-ended questions, which require accurate numerical calculation and application of tax rules. We hope that this paper provides a means and a standard to evaluate the efficacy of current and future LLMs in the tax domain.

Demander à l'IA

Bookmark

View Full Paper