BERT, XLNet or RoBERTa: The Best Transfer Learning Model to Detect Clickbaits

Clickbait can be a spam or an advert which more often provides a link to a commercial website, or it might be a headline to a news media website which makes money from page views by providing eye-catchy headlines with deceptive news. This paper focuses on the latter one to identify clickbaits that use news headlines to publish news items in Twitter. In this work, we aimed to use Transfer Learning approaches by adding various configuration changes to the existing models in order to detect clickbaits. Based on author’s knowledge, this is the first attempt to adapt Transfer Learning to classify Clickbaits. The analysis in this work are mainly focused on three main models: BERT, XLNet and RoBERTa.We fine-tuned these models by integrating novel configuration changes to the default architecture such as model expansion, pruning and data augmentation strategies. We used Webis Clickbait Challenge 2017 dataset to train our models and it was introduced to evaluate the level of clickbait of a Twitter post. The best performed model at this competition is considered as the benchmark for this research. Our analyses were mainly focused on eight different scenarios after applying several fine-tuning approaches and model configuration changes. The results shown that, our proposed Transfer Learning approaches outperformed the considered benchmark. In our experiments, the best performed Transfer Learning model is RoBERTa with the integration of an additional non-linear layer to the output tensors extracted from hidden outputs. For binary classification, this configuration has achieved 19.12% more accuracy in compared to the benchmark model. There is no significant improvement when models expanded by adding extra RNN layer(s). Apart from that, we experimented with another labelled clickbait dataset (Kaggle clickbait challenge) to explore the performance of our fine-tuned models under different scenarios.