What question did this study set out to answer?

This study aims to evaluate the effectiveness of LoRA-based fine-tuning on Qwen3 models for bilingual legal question answering in Kazakhstan.

July 8, 2026Open Access

Fine-Tuning Qwen3 Models for the Legal Domain of Kazakhstan: A Comparative Study of LoRA-Adapted Models for Bilingual Legal Question Answering

Key Points

This study aims to evaluate the effectiveness of LoRA-based fine-tuning on Qwen3 models for bilingual legal question answering in Kazakhstan.
Utilized a bilingual dataset of 63,114 question-answer pairs (76.2% Russian, 23.8% Kazakh) across 11 legal domains.
Models were assessed using automated metrics like BERTScore and blind expert evaluations.
Compared the performance of fine-tuned models against untuned base models and commercial equivalents.
Fine-tuned models achieved BERTScore F1 of 89.6–90.2%, outperforming base models (82.2–83.1%).
Fine-tuned models showed better semantic similarity scores than GPT-4o (87.2%) and lower hallucination rates (27–29% vs. 83–90%).
Expert assessment resulted in completeness scores of 4.28/5 for fine-tuned models compared to 1.95/5 for base models, indicating significant quality improvement.

Abstract

This paper reports a systematic study of low-rank adaptation (LoRA) -based fine-tuning applied to Qwen3 language models (4B, 8B, and 14B parameters) for the task of legal question answering within the jurisdiction of the Republic of Kazakhstan. The bilingual dataset comprises 63, 114 question–answer pairs (76. 2% Russian, 23. 8% Kazakh) covering 11 legal domains. Models are evaluated through both automated metrics (BERTScore, citation accuracy, and hallucination rate) and blind expert assessment by a panel of two practising legal experts. Key findings: (1) all fine-tuned models reach BERTScore F1 close to 90% (89. 6–90. 2%) versus 82. 2–83. 1% for untuned base models; (2) fine-tuned models outperform GPT-4o (87. 2%) and GPT-4o-mini (86. 7%) on semantic similarity while exhibiting far lower hallucination rates (27–29% vs. 83–90%) ; (3) blind expert assessment confirms the advantage of fine-tuned models, with panel mean completeness scores of 4. 28/5 versus 1. 95/5 for base models (quadratically weighted Cohen’s κ = 0. 80–0. 95 across rating dimensions, indicating substantial to almost perfect inter-rater agreement) ; and (4) we identify a practical scaling plateau: paired Wilcoxon tests (n = 500) detect statistically significant but practically small differences across the 4B, 8B, and 14B fine-tuned variants (largest mean gap 0. 67 pp on BERTScore F1; Cohen’s |dᵦ| ≤ 0. 34), gains too small to justify the 3. 5× parameter increase. These findings show that parameter-efficient adaptation of compact open-source models can match or exceed commercial LLMs for specialised legal QA in a low-resource bilingual context. We note one scope restriction: three domains (administrative, criminal, and housing law) are represented only in Russian, so the model is not validated for Kazakh language queries in these areas.

Read Full Paperexternally

Perguntar à IA

Bookmark

View Full Paper