Naming a Crawler Unblocks It: Rule Shadowing in robots.txt Across 12,828 Popular Websites

Under the Robots Exclusion Protocol (RFC 9309), a crawler obeys only the group of rules that names it, and falls back to the User-agent: * group only when no group does. A site that adds a group for one crawler therefore removes that crawler from every rule written for crawlers in general, unless the new group repeats them. As sites add groups for AI crawlers, this interaction is easy to trigger and invisible on inspection: each line is correct, and the file reads as intended. We measure it on a stratified sample of 20,000 origins from the Chrome UX Report top-million list, of which 12,828 served a readable robots.txt. Among the 2,737 sites that both restrict part of the site for all crawlers and name at least one of 26 search or AI crawlers, 1,061 (38.8%, 95% CI 37.0–40.6%) give at least one named crawler access to paths their general rules block; 707 of these include a search crawler. Where a site names Googlebot, 63.0% of such groups drop rules the general group sets, and 61.8% of affected crawler–site pairs drop every one of them. Three patterns account for the cases: groups that allow everything (552 sites), copies of the general rules that have drifted (357), and groups meant to add restrictions that instead replace them (229). An independent parser agrees on 1,061 of 1,062 flagged files, and all 30 files in a random sample were confirmed by hand. The exposed rules most often cover query-parameter pages, administration and login paths, and internal search. We state the limits plainly: a file cannot reveal intent, the sample was read once from one network, and only 26 crawler names were evaluated. We release the measurement code and the exact sample definition. Source, data and code: https://github.com/emmanuelorta/crawlcheck-core/tree/2770804e4ff0372e070c6fcfaadcfdaef9727991/paper/robots-shadowing.