LLM4Gov: A Privacy-Preserving Approach to Teacher-Student Fine-Tuning of Distilled LLMs for the Public Sector

Large Language Models (LLMs) have revolutionized natural language processing, but their reliance on extensive computational resources and proprietary APIs poses significant challenges for public sector applications. Government agencies often face legal constraints, such as GDPR and LGPD, preventing the use of external LLMs when handling sensitive data. Even when compliance is met, the financial burden of deploying large-scale models remains a major barrier. To address these challenges, we present LLM4Gov, a privacy-preserving computational tool designed to fine-tune distilled LLMs for government-related tasks while minimizing infrastructure costs. LLM4Gov follows a structured teacher-student learning pipeline, where a lightweight anonymization module first removes personally identifiable information (PII) before any interaction with an external LLM. A teacher LLM then generates task-specific instructions from the anonymized dataset, and a distilled student LLM is fine-tuned using Low-Rank Adaptation (LoRA) and quantization, enabling deployment on resource-constrained environments. Our experimental results show that LLM4Gov consistently outperforms competitive distilled LLMs, achieving higher accuracy while preserving privacy and interpretability.