Exploiting latent contextual knowledge for high-performance lightweight large language models: A framework for children’s language

Large language models (LLMs) are effective at generating and simplifying text, but they lack the ability to adjust the level of simplification and associated semantics for specific audiences, such as children, and implementing these adjustments is expensive. This limitation leads some approaches to avoid actively utilizing LLMs and instead rely on human intervention for effective simplification, which restricts their scalability and practical application. To resolve this challenge, we propose a framework for developing a lightweight LLM specifically designed to automatically transform complex texts into a format suitable for children, ensuring appropriate vocabulary, semantics, sentence structure, and simplicity. Central to our framework is a novel method called Teacher–Student Task Separation inspired by offline distillation, which smartly addresses the cost issue of directly tuning LLMs for this purpose. We use an LLM as a teacher model to generate children’s language based on prompt engineering with a beginner’s dictionary and children’s stories, without any learning. Then, we train a lightweight LLM as a student model using an advanced dictionary and the data generated by the teacher model. This method, referred to as offline distillation through data, employs task separation to allow the lightweight student model to efficiently capture latent contextual knowledge tailored for children’s language and even outperform the LLM, the base for the teacher model, in quantitative and qualitative evaluations including a new target-specific comprehension metric. Our work provides a cost-effective solution using LLMs to enhance children’s comprehension of diverse content and has the potential for adaptation to other purpose-specific LLMs. 1