Answers are grounded in this paper and cite their sources.
Enhancements in a pretrained Thai language model improved understanding of foreign vocabulary in tasks, suggesting better outcomes.
Although WangchanBERTa has become the de facto standard in transformer-based Thai language modeling, it still has shortcomings in regard to the understanding of foreign words, most notably English words, which are often borrowed without orthographic assimilation into Thai in many contexts. We identify the lack of foreign vocabulary in WangchanBERTa’s tokenizer as the main source of these shortcomings. We then expand WangchanBERTa’s vocabulary via vocabulary transfer from XLM-R’s pretrained tokenizer and pretrain a new model using the expanded tokenizer, starting from WangchanBERTa’s checkpoint, on a new dataset that is larger than the one used to train WangchanBERTa. Our results show that our new pretrained model, PhayaThaiBERT, outperforms WangchanBERTa in many downstream tasks and datasets.
No takes yet. Share an insight, caveat, or question.
Sriwirote et al. (2025) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: