Evaluating Publications' Keywords in Computer Science Education Research: A Bibliometric NLP Approach

In the rapidly growing field of computer science education research (CSER), the collection of publication keywords represent the knowledge landscape of the research domain. This helps stakeholders to contextualize existing research topics, identify the strategic direction for future works, and shape the culture of this growing community. For the research studies published in CSER, the use of proper keywords will help to improve publication visibility in research networks. Publication visibility is defined as the share of traffic a study receives based on its ranking from Search Engine Result Pages (SERPs), which can be gained through indexing relevant keywords to help search engines to find the related studies. It is important to acknowledge that analyzing keyword selections in the context provided by the publication abstract, enables us to determine the keywords’ relevance to the research topics. However, evaluating the selection of publication keywords is an important but lesser-addressed issue. In this work, we utilized a bibliometric approach to scrutinize publication keywords in CSER publications from 2015 to 2020. We gathered the titles, abstracts, and keywords collectively from studies published in two major CS education conferences and journals. We also extracted the strategic directions from their mission statements. By applying the Natural Language Processing (NLP) technique, known as Term Frequency-Inverse Document Frequency (tf-idf), we aimed to answer: 1) What are the most prevalent research foci represented in CSER through the publication keywords? 2) Are publication keywords in alignment with the CSER publications’ stated strategic directions? 3) Do selected keywords in CSER publications adequately represent the research topics presented via publication abstracts? Our preliminary results suggest that, as expected, the most prevalent research foci represented by the most commonly used keywords of CSER were on teaching and learning and broadening participation, which is in alignment with the corresponding strategic directions. Our analysis suggests that the CSER community is putting efforts to formulate a diversified culture through teaching and learning for inclusivity. However, our results also suggest that there is a misalignment between keyword identification by authors and the topics presented in the abstracts. With our work, we hope to motivate scholars to carefully evaluate and select the keywords that will be used in the future for indexing publications to improve the research topic relevancy and publication visibility for broader impact.