Quantifying Concept Definition Heterogeneity in Academic Texts: Insights into Variability in Conceptualization

Academic texts often define a given concept using words. Across texts, definitions may represent the same or different meanings of the given concept, especially for concepts in interdisciplinary fields, such as cognitive science. Understanding the variability of definitions and the reasons for this variability may facilitate the replicability of science: variations in concept definitions may indicate or provoke a heterogeneous understanding of the concept, leading to differences between studies in the methodology and, subsequently, the results. We developed proxy measures of the variability in definitions of each concept using information theory and natural language processing (i.e., entropy, KL divergence, and lexical diversity). We collected and analyzed 2212 definitions of 216 concepts associated with "open scholarship" as a use case study. We found strong correlations between the entropy-based proxy measures and the number of definitions and unique words yielded by concepts. However, we found no correlation between lexical diversity and the entropy-based proxies. This indicates that concepts that yield definitions with different wordings in the literature (high on lexical diversity) do not necessarily reflect an inconsistent understanding of concepts’ meaning across contexts (high on the entropy-based measures). The proposed methodology can help identify concepts that yield heterogeneous definitions in academic texts and concepts that are consistently defined in a semi-automated manner.