Rumour stance classification adaptation under domain shift: a case study of rumours in Ireland

Considering a conversation initialised by a rumour-introducing source post on social media, rumour stance classification aims to identify the opinion (e.g. agree or disagree) of its replies towards the source post. Methods for automatic rumour stance classification have yielded high performances; however, none of them have been evaluated on a dataset that is substantially different from the data they were trained and tested on, putting their reliability on real-world applications into question. This paper releases a novel rumour stance classification dataset (ISLES), focusing on online rumours spread in Ireland, that features substantial differences from commonly used datasets over topic, lexicon and stance distribution. We evaluate the top-performing models trained with RumourEval data on ISLES and observe considerable performance degradation when compared to the performance on RumourEval test sets. Further, we seek to adapt a RumourEval-trained model to the domain of rumours spread in Ireland, in the absence of both labelled RumourEval and Irish-domain rumours data. By leveraging synthetically generated Irish-domain rumour stance data with large language models, our proposed self-training framework for domain adaptation successfully boosts the performance of the RumourEval-trained model by nearly 127% on ISLES.