Report AI Crawler Traffic in Two Columns, Declared and Verified

Most counts of AI crawler traffic assign a request to a crawler by its User-Agent header. The client writes that header, so the count shows what clients said about themselves, not who they were. I reanalysed fifteen days of complete access logs from one origin server behind a CDN. The logs hold 345,808 requests, and 46,155 of them carry one of 22 published crawler tokens. For 35.7% of the token-bearing requests, the client connected to the origin directly. Not one of the 11,276 direct requests under tokens with published address ranges came from those ranges. The direct traffic also behaves differently from the traffic that came through the CDN. It never asked for robots.txt or the sitemap, and per request it used about three and a half times as many distinct User-Agent strings. The CDN route is not proof either. A token that Google says it never sends as a User-Agent came through the CDN 293 times. At this origin, no single request could be confirmed by address, because the CDN-routed requests carry the CDN’s address in the log. I propose that reports of crawler traffic give separate counts for declared, route-consistent, verified and unverifiable requests. They should state which verification was possible. They should never present the declared count alone as crawler activity.