Miasma: an open-source, lightweight countermeasure against unauthorized AI web scraping

The widespread adoption of large language models has greatly increased demand for publicly accessible web data, resulting in increased web scraping across the internet. The resulting bandwidth and compute costs fall on site operators rather than on the parties collecting the data, with scientific databases and journals among those reporting service disruption [1]. Existing mitigations such as rate limiting, CAPTCHAs, and robots.txt rules can discourage well-behaved crawlers, but impose no cost on scrapers that choose to ignore them. Deception-based alternatives have shown promise, though prominent examples such as Cloudflare’s AI Labyrinth remain tied to enterprise platforms [2]. Separately, recent work suggests that small quantities of poisoned content can compromise a model [3], and that continual training on low-quality content measurably degrades model performance [4]. Miasma is an open-source tarpit that brings a deception-based defense within reach of small operators. Implemented in Rust, it runs as a standalone service or as an embeddable library, trapping unwanted crawlers in a recursively generated network of links whose pages carry deliberately poisoned content. Miasma does not perform bot detection itself. Rather, it is paired with an upstream mechanism such as hidden links or an existing detection tool, and its poisoned content is drawn from a configurable external source. Instead of blocking requests outright, Miasma reduces the value of what is collected while keeping its own operational footprint small. To that end, Miasma uses asynchronous request handling on the tokio runtime, a bounded in-flight request cap that places a hard limit on memory consumption, and a custom HTML templating system that streams page chunks without allocating intermediate strings. Miasma is able to serve 100 sustained in-flight requests while using roughly 50 MB of memory. Across six months of deployment on a small portfolio site, Miasma served 210 million requests and 1.29 TB of poisoned content. As of October 2026, the project has reached 1,200+ stars and 6 contributors on GitHub, with 6,500+ Docker pulls, and is distributed under GPL-3.0 via Cargo, Docker, and community-maintained packages.