KaEnLandetector: Rule-Based Language Annotation and Transformer-Based Language Detection for Kannada-English Code-Mixed Text

In multilingual societies, people tend to mix multiple languages for communication. This phenomenon is known as code-mixing or code-switching. This is visible more on social media platforms and e-commerce websites to share their opinions and feelings. Automatic language detection has become a crucial step in language processing for subsequent tasks. This study focuses on detecting the language of Kannada–English code-mixed sentences at the word level. The dataset is prepared by annotation based on rules prior to the detection. The transformer-based approach is applied using BERT and its variants. The models label words as English, Kannada, Mixed, Named Entity, and Universal for the given code-mixed texts. The highest accuracy (98%) was obtained using XLM-RoBERTa .