SQL Query Evaluation with Large Language Model and Abstract Syntax Trees

SQL stands as the foundational language for data analysis and manipulation, playing a pivotal role in the database learning process. Proficiency in SQL is essential for students seeking to excel in data-related fields. However, the conventional approaches to assessing SQL queries rely heavily on manual grading, and the automated assessment tools are usually producing only binary decisions for the submitted queries. Our primary research objective is to develop effective methods for evaluating the quality of the SQL queries. To meet this objective, we introduce two approaches: structure-based analysis and evaluation by an instruction tuned large language model (LLM). The first approach deconstructs queries into Abstract Syntax Trees (AST) and employs cosine similarity to assess student submissions. The second approach utilizes a pre-trained LLM: FLAN-T5, fine-tuned for predicting the quality of student submissions. These methodologies are tested on a SQL dataset, and our experimental findings evaluate against a grading rubric with categories ranging from "good" to "unacceptable". The experimental results demonstrate that we can enhance the grading efficiency by applying these approaches and illustrate the ability of utilizing LLM to classify the assessed SQL statements more accurately. In addition, this research contributes to Computer Science (CS) education by integrating these approaches into our team's automated SQL statement assessment tool, improving the learning experience and evaluation process.