Key points are not available for this paper at this time.
As generative AI continues to evolve, it becomes increasingly important for site owners to effectively communicate their conditions and preferences to web agents to maintain data sovereignty. This necessity underscores the importance of an ecosystem where the technical means to prevent unauthorized data mining and to set conditions on the usage of web resources are readily available. Our research focuses on the temporal development of such technical content control methods, examining two primary mechanisms: the regulation of web robots via the Robots Exclusion Protocol and the semantic annotation of web documents with licensing information. Through a longitudinal study, we analyze the implementation and recent modifications of robots.txt files, robot directives (such as noindex, nofollow, etc.), and license-related HTML annotations. This study is driven by the growing awareness among site owners regarding the control over their content in the face of the progression of AI, highlighting the critical need for effective web content control strategies to protect and appropriately manage the wealth of texts, images, videos, and other content populating the internet.
Dinzinger et al. (Sun,) studied this question.