New methods are necessary to understand the presence and impact of generative AI on academic writing in aggregate. Available AI detection tools check for complex, probabilistic signals, such as the likelihood of specific word sequences. These methods can be circumvented through repeated paraphrasing, and are costly to run in aggregate. We propose an alternative approach: tracking changes in the distribution of established style metrics such as Flesch-Kincaid Reading Ease, passive-voice ratio, and more. In a a pilot study analyzing 31,624 publicly available abstracts from 2004 through 2024, we find distinct trends that coincide with the emergence of LLMs. Indicating these metrics can provide a lightweight and explainable means for detecting the systemic use of AI across large bases of academic writing. In this paper, we detail these metrics and trends to determine their viability for tracking general writing change in an academic setting.
Silbiger et al. (Tue,) studied this question.