Performance evaluation of LLMs in the Text-to-SQL task in Portuguese

Context: The rising need for consulting data in industry and academic contexts has fueled Text-to-SQL development, where natural language queries are translated into SQL, making data access easier. Problem: Most research focuses on English Text-to-SQL, leaving Portuguese—a language spoken by over 260 million people— underrepresented, creating challenges for organizations reliant on accurate data retrieval in Portuguese. Solution: This study evaluates the effectiveness of various Large Language Models (LLMs) on Portuguese Text-to-SQL tasks using a validated translation of the Spider benchmark. IS Theory: This research applies Task-Technology Fit (TTF) Theory to assess how well LLMs meet the needs of Portuguese Text-to-SQL tasks. TTF evaluates the match between LLM capabilities and task requirements, including language understanding, schema recognition, and SQL generation. Proper alignment is key for effective data retrieval, particularly for Portuguese-language applications in organizational decisionmaking. Method: A comparative analysis of seven LLMs—tested on both Portuguese and English Spider benchmarks—was performed. Exact Match (EM) and Execution Accuracy (EX) metrics measured performance, and a zero-shot prompting approach maintained consistency. Results Summary: Larger LLMs and specialized code models excelled, showing less performance variance between Portuguese and English tasks. Generalist models, however, produced verbose outputs, which may limit practical use in production systems. Contributions and Impact on IS: This research establishes baseline Portuguese Text-to-SQL metrics and insights into language adaptability in LLMs, offering guidance for organizations seeking Portuguese-language data solutions. By bridging language gaps, it advances data-driven practices and fosters growth in Portuguese-speaking regions.