What question did this study set out to answer?

This research aims to evaluate Llama 3 and other models for effective sentiment analysis in code-mixed languages, focusing on Banglish.

March 26, 2026Open Access

Sentiment Classification in Low-Resource, Code-Mixed Languages: Comparative Study of Llama 3 and Other Large Language Models

Key Points

This research aims to evaluate Llama 3 and other models for effective sentiment analysis in code-mixed languages, focusing on Banglish.
Conducted systematic evaluation on Llama 3 for sentiment analysis of Banglish.
Examined five adaptation strategies: zero-shot prompting, few-shot learning, two-step pipeline, fine-tuning, model ensembling.
Utilized the Bengali_Banglish_80K dataset containing 11,673 posts for experiments.
Fine-tuning Llama 3 achieved the highest accuracy at 66.87%.
Zero-shot performance for Llama 3 was 43.80%, and few-shot performance was 49.14%.
Compared to other models, fine-tuned Llama 3 outperformed GPT-3.5, GPT-4, and Claude 3.5 in zero-shot settings.

Abstract

Large language models have seen limited evaluation on code-switched languages, despite the increasing use of such linguistic patterns on social media platforms. This study presents a systematic evaluation of Llama 3 for sentiment analysis of "Banglish", the informal blend of Bengali and English widely used online in Bangladesh. We examine five adaptation strategies: zero-shot prompting, few-shot learning, a two-step translation and analysis pipeline, fine-tuning, and model ensembling. Our experiments, conducted on 11, 673 posts from the BengaliBanglish₈0K dataset, show that fine-tuning Llama 3 delivers the best performance, achieving an accuracy of 66. 87 percent. This result outperforms its own zero-shot (43. 80 percent) and few-shot (49. 14 percent) baselines, as well as the zero-shot results of GPT-3. 5 (55. 90 percent), GPT-4 (65. 15 percent), and Claude 3. 5 (47. 68 percent). The dual-phase pipeline achieved 48. 07 percent accuracy, and the ensemble method reached 66. 78 percent, offering slight improvements but falling short of the fine-tuned model. We release our best-performing model, samiur-r/BanglishSentiment-Llama3-8B, to support further research. These findings highlight the effectiveness of task-specific fine-tuning in low-resource, code-mixed settings and emphasize the need for more comprehensive code-switching datasets and pre-training strategies tailored to linguistically diverse communities.

AIに質問

Bookmark

View Full Paper