What the Web Shows to AI: Declared versus Enforced Access for Training, Search and User-Triggered Agents

Studies of AI crawler blocking compare an AI user agent's page to a browser's, yet such a request claims an identity it cannot prove. We request the Tranco top 10,000 and a sample of smaller sites with 10 AI agents, neutral controls and spoofed Googlebot and Bingbot, from a residential and a datacenter vantage point. Robots.txt rules and server refusals mostly fall on different sites. Beyond the rate at which sites refuse a spoofed search engine, only training crawlers are refused more often. Where a spoofed user-triggered agent was refused, the real ChatGPT failed on 51.5 % of checks; Claude was mostly stopped by its own fetch tool. HTTP 402 to AI agents carries no price. Pre-registered on OSF: https://doi.org/10.17605/OSF.IO/PRWNGCode, aggregate results and hashed per-site data: https://github.com/Nu1111111111/ai-web-measurement