Fine-Tuning a Compact Language Model for Open-Domain Conversational AI A Case Study Using DistilGPT2

This paper presents a case study on fine-tuning DistilGPT2, a distilled transformer language model with 82 million parameters, for open-domain conversational tasks using the OpenAssistant Conversations (OASST1) dataset. We document the complete experimental pipeline including data preprocessing, model configuration, training dynamics, and qualitative evaluation. The model achieved a validation loss of 1.6963 after two training epochs, demonstrating that the model successfully learned conversational turn-taking patterns. However, generation examples reveal persistent challenges including response repetition and factual inconsistencies, which are attributable to the model's architectural constraints rather than the quality of the training corpus. This study provides practical insights for researchers working with resource-efficient transformer fine-tuning in conversational AI applications. Keywords: DistilGPT2, conversational AI, fine-tuning, transformer models, open-domain dialogue, resource-efficient NLP