Gifted SLM-as-a-Judge with Attention Steering: Fact-Checking and Detailed Topic Extraction from Academic Literature

This study proposes the development of an academic domain-specialized Small Language Model (SLM), designed to be used in combination with a general-purpose Large Language Model (LLM), with the objective of constructing a knowledge graph through the highly accurate extraction of detailed technical elements, such as research objectives, methodologies, and experimental targets from scholarly articles. The proposed academic domain-specialized SLM is developed through continuous pretraining using collections of survey papers within the target domain and is further optimized to specialize in extracting domain-specific technical elements through attention steering techniques. In the knowledge graph construction process, the SLM is employed as a fact-checking mechanism for outputs generated by a general-purpose LLM. In this paper, we conducted experiments on extracting technical elements in two fields: Fake News Detection in Computer Science and Bibliometrics in Science and Technology Management and confirmed that the proposed academic domain-specialized SLM outperforms general-purpose LLMs in terms of the accuracy, stability, and importance of the extracted information. Future work includes constructing a knowledge graph by integrating the proposed SLM with general-purpose LLMs.