Building a canary site to catch AI training scrapes
The question nobody can answer from their server logs
Did my website end up inside an AI model? Everyone who publishes on the web now has a version of this question, and almost every answer is a shrug. You can block crawlers in robots.txt and hope they listen. You can look for GPTBot in your access logs and learn that something fetched your pages. Neither tells you whether your words are now weights.
So we built an instrument to find out: a dedicated site whose only purpose is to be scraped, seeded with traceable canaries, fronted by an edge layer that fingerprints every visitor and stamps every page it hands to a bot. The site and the canary values stay secret. The method doesn’t need to, and that’s what this post is about.
Two problems, not one
Detecting a crawl and detecting ingestion are different problems with different evidence.
The crawl happens at your server. Evidence is a user-agent string, an IP you can check against the vendor’s published ranges, a TLS fingerprint, header order, whether the client fetched CSS and images or just HTML, whether it honoured robots.txt. All of it is observable in real time, and all of it can be faked by a determined scraper.
Ingestion happens months later, inside a training pipeline you’ll never see. The only evidence is behavioural: does a model know something it could only have learned from your pages? That requires planting facts that exist nowhere else, and then asking.
The design goal was to connect the two. A canary that surfaces in a model should trace back to one specific fetch, with that fetch’s full fingerprint. That turns “an AI company scraped me” into “this request, from this IP, claiming to be this crawler, at this time, received this exact sentence, and here it is coming out of the model.”
First, find out what survives the extractor
Before writing a line of the site we ran one experiment, and it decided the whole design. Training pipelines don’t feed raw HTML to a model. They run it through a boilerplate remover, most often trafilatura (which FineWeb, RefinedWeb and Dolma all use), to keep the article and throw away the nav, footer, sidebar and metadata. A canary the extractor drops never reaches a model, however cleverly hidden.
We generated 20 realistic article pages across three layouts (blog, ad-heavy news, wiki), embedded 26 candidate canary channels on each, and ran them through 10 extractor configurations. 520 canaries, scored on whether the exact phrase survived.
| Channel | Verbatim survival | Verdict |
|---|---|---|
| Aside sentence inside a body paragraph | 100% | Build on it |
| Word-choice variants in the prose | 100% | Build on it (invisible, needs statistical probes) |
| H2 subheading | 100% | Build on it |
| Pull quote | 100% | Build on it |
| Link anchor text, list item, table cell, reference entry | 80–90% | Fine for the pipelines that matter |
| CSS-hidden text (display:none, offscreen, white-on-white) | 70–80% | Survives, but it’s cloaking; don’t |
| Figure caption, byline, author bio | 40–60% | trafilatura strips all three |
| Image alt text, title tag, meta description, sidebar, footer | 10–20% | Dead |
| JSON-LD, HTML comments, data attributes | 0% | Dead |
Two results were surprising. The most natural hiding places, bylines, captions, author bios, are the ones trafilatura removes most reliably. And CSS-hidden text sails through: extractors walk the DOM and never apply a stylesheet. It’s still the wrong choice for a real site because search engines treat it as cloaking, but the folk wisdom that “hidden text gets stripped” is false at the extraction stage.
The layouts made no difference. Every channel scored the same across blog, news and wiki, so the result is about the extractors, not the page.
Designing a canary a model will remember
A canary has to satisfy two conflicting requirements. It must be unique enough to trace, and it must look enough like a fact that a model will bother to memorise it. Models don’t learn hex strings. They learn that a person did a thing in a place in a year.
So the canary is an encoding. A 48-bit ID is split into four 12-bit fields, and each field indexes a 4,096-word synthetic vocabulary: a given name, a surname, a place name, an event. All four vocabularies are built from syllables, so every entry is plausible and none is a real person or town. Two more fields are derived from a hash of the ID: a check word drawn from 256 “archive” nouns, and a year between 1800 and 1927.
The six fields render into one of five sentence frames. Stripped of real values, a frame looks like this:
The {event} of {year} at {place} was first recorded by {person} in the {check}.
It reads as a footnote in a parish history. It carries 48 bits of identity plus 15 bits of redundancy.
Decoding runs the other way. A scanner reads any text, a model’s answer, an extracted page, a dataset shard, and looks for vocabulary words that co-occur within a short window. It reassembles the ID and accepts it only if the check word and the year both match what the hash predicts. A spurious match passes about one time in 32,000; on 15,000 words of vocabulary-dense junk it produced zero false positives. The Python and JavaScript encoders are byte-identical, tested on random IDs, because the site encodes in JavaScript at the edge and the lab decodes in Python.
A second, quieter channel rides alongside: eight pairs of interchangeable words in the prose (“began” or “commenced”, “large” or “substantial”) encode eight more bits by which word was chosen. Nobody can see it, and no model will recite it, but membership-inference probes that measure how surprised a model is by a text can pick it up.
Four tiers, four hypotheses
Uniqueness and memorisation pull against each other. A sentence that appears once in a trillion-token corpus almost never surfaces; a sentence that appears on every page is memorable but tells you nothing about who fetched it. The answer is a hierarchy, with each tier a separate hypothesis and its own base rate.
| Tier | Who receives it | Rotates | What a hit proves |
|---|---|---|---|
| Site | Everyone, humans included | Never | The site was ingested at all |
| Vendor | Each AI crawler, keyed on verified identity | Never | Which company’s crawler did it (spoofed user-agents get their own ID) |
| Window | Each AI crawler | Weekly | When the ingestion happened |
| Request | Each AI crawler, per fetch | Every request | The exact fetch, with its full fingerprint |
Humans and search engines only ever see the site-tier sentence, so the page they get is stable, cacheable and identical for everyone. There is no cloaking in the search-engine sense: the canary is in the page for all visitors. Bots get three more sentences and a no-store cache header.
The request tier will rarely be memorised. That’s expected. The site and vendor tiers do the heavy lifting on probability; the request tier is the forensic trace for when something does surface.
The bait site
The site itself is 300 generated articles of plausible local history: the wool trade, the tramway, the flood of the lower valley, each seven paragraphs of vestry minutes and county-gazette prose, built with Astro and deployed as static HTML. Every word of it, and every static canary in it, derives from one seed. The repository contains no canary values; the build generates them, and the same seed regenerates the manifest locally for decoding later.
The pages are sorted into exposure cohorts, so the data later says not just whether the site was crawled but how deep the crawler went:
| Cohort | Pages | Reachable via |
|---|---|---|
| Linked | 100 | Navigation, homepage, index pages, cross-links, sitemap |
| Deep | 80 | Links from other articles only, plus the sitemap |
| Sitemap-only | 50 | sitemap.xml and nothing else |
| Orphan | 40 | Nothing. The URLs exist only in the private manifest |
| Private | 30 | A path disallowed in robots.txt, linked only by an off-screen link in every footer |
The last cohort is the honeypot. A crawler that follows an invisible link into a disallowed directory has broken two rules at once, and the request gets logged with that flag.
On top of the cohort canaries sits a repetition ladder: static canaries placed on 300 pages, 100, 10 and 1, plus twenty singletons. When probes run against models later, the hit rate at each rung gives a memorisation curve, which is data nobody currently publishes.
The edge: where every fetch becomes a row
The site is static. Everything dynamic happens in an edge middleware that sits in front of it and runs on every HTML request.
One request, six steps:
- Fingerprint. Read the user-agent, IP, country, JA4 TLS fingerprint, header order, Accept headers, cookies and referer.
- Classify. Match against the crawler list and, where the vendor publishes IP ranges, mark the request verified, spoofed or claimed. Scripted clients fall into bot-unknown; everyone else is human.
- Assign IDs. Derive the site, vendor, window and request tier IDs for this fetch.
- Inject. Humans and search engines get the page as built. Bots get three more canary sentences appended to the body.
- Cache. Bot responses go out with no-store, so every fetch gets its own request-tier canary.
- Log. Write one row to the request log.
Classification uses a list of 25 known crawlers. For vendors that publish IP ranges (OpenAI, Google, Microsoft, Perplexity) the request’s IP is checked against them, so a crawler is logged as verified, spoofed, or claimed when the vendor publishes no ranges. Anything else with a scripted-client tell, a bot-ish user-agent, no Accept-Language, a wildcard Accept, lands in bot-unknown, which still receives the full canary set. The remainder is human.
Injection is deliberately dumb. The middleware finds the paragraphs with at least 200 characters of visible text and appends one canary sentence to each of the second, third, fourth and fifth. That rule works on any DOM, which means the same middleware can front a real site, not just the bait.
The log row carries everything: timestamp, path, honeypot flag, session, classification, the four tier IDs, user-agent, IP, country, JA4 TLS fingerprint, header order, Accept headers, cookies, referer. The canary sentence itself is never stored; the ID is enough to regenerate it.
What showed up in the first hour
Nobody had linked to the site. It wasn’t in any sitemap submission or search index. Within five minutes of the domain going live, requests arrived from Ashburn, London, Frankfurt, Phuket, New York and Santa Clara, several with empty user-agents, one spoofing a 2016 Safari.
That’s the certificate-transparency crowd. Issuing a TLS certificate is a public event, logged to CT, and a population of scanners watches those logs for new hostnames. They are not AI crawlers, but they are a useful calibration: the classifier put every one of them in bot-unknown, each received a request-tier canary, and each has a row.
The very first row in the database was more interesting. It was an AI vendor’s user-triggered fetch agent, not a training crawler but the thing that goes and reads a page when a user asks a chatbot about a URL. It arrived because we asked an assistant to check the deployment. It was classified correctly, received all four tiers, and its JA4 fingerprint was recorded on a free hosting plan, which we hadn’t expected.
It was also a lesson. Any time anyone asks an AI assistant to look at the site, that fetch becomes a vendor-tier row and the page’s canaries enter a model’s context window. Which is a form of exposure the experiment has to account for, and a reason to never discuss the site with a model outside a scored probe.
What we got wrong along the way
Four things broke, and each one would have quietly invalidated the experiment.
Curly quotes. The markdown renderer converted the apostrophe in one event name to a typographic quote. One frame stopped decoding. The fix was to disable smart punctuation at build and normalise quotes in the decoder, but it’s a reminder that the text a model sees has been through several hands, and the decoder has to be forgiving about every one of them.
Clean URLs. The static build produced about.html, and the host served it only at /about.html. Every one of the 300 article URLs in the sitemap was a 404 for the first twenty minutes. A sitemap full of dead links would have sent every crawler away. One config line fixed it; the lesson is to fetch a deep page, not the homepage, when verifying a deploy.
The database driver. The first production build logged nothing. The edge runtime raised an exception on every request because the pinned driver version predated the query method the code used. Runtime logs caught it within a minute. Without them, the site would have run for weeks recording nothing.
Our own earlier claim. Before the survival experiment, the plan assumed CSS-hidden text would be stripped by extractors. It isn’t. A design built on that assumption would have hidden canaries in the wrong places. Measuring before building is the whole reason the survival matrix came first.
The rules the experiment runs on
Most of the design is code. These rules aren’t code; they’re discipline, and breaking any one of them fails silently. Nothing errors. The data just stops meaning what it says. They’re ranked by how much they would cost.
Rules that, once broken, can’t be repaired
1. The site is never tied to anything public. No link from this post, this site, or anywhere that describes the method. Inbound links come from places that don’t say what the site is. A crawler that arrives because it read about the experiment isn’t a control.
2. No model sees the site outside a scored probe. Don’t ask an assistant to check it, don’t paste a page into a chat, don’t ask a model to debug it. Every one of those is a vendor fetch, or canaries in a context window, and it’s exactly the contamination the probes are meant to rule out. Debugging goes through curl and server logs.
3. The seed is never lost. Every canary on the site derives from one seed. Without it, a hit can’t be decoded and the request log is just a list of visitors. It’s backed up offline, in more than one place.
4. The seed and the canary values are never published. Not in the repository, not in a post, not anywhere a crawler can reach. A published canary is a dead canary.
5. The encoder is frozen. The vocabularies, sentence frames, word-pair channel and hash don’t change after launch. Change one and everything already served stops decoding. If a second version is ever needed, it gets a version number, and the first stays decodable forever.
6. The content is frozen. Same seed, same 300 articles, same cohorts, same repetition ladder. The dose-response curve only means something if the doses don’t move.
Rules that, once broken, quietly corrupt the data
7. Hidden pages stay hidden. The orphan and private URLs are never linked, submitted or visited from anywhere. One stray link turns an orphan into a deep page, and the measure of how deep a crawler went is gone.
8. Nothing stands between the crawlers and the site. No bot protection, no AI-crawler blocking at the host, no robots.txt edits mid-run. A blocked crawler is indistinguishable from one that never came, and a policy change in the middle confounds the time windows.
9. The log is watched. Runtime errors get checked after every deploy, and storage and plan limits get checked on a schedule. A full database looks exactly like a quiet month.
10. Nothing transforms the text at build. No smart punctuation, no typography plugins, no minifier that touches prose. The curly-quote bug was one of these.
11. The domain never lapses. Auto-renew stays on. A new owner would inherit the crawl history and could serve anything.
Rules for reading the results
12. No probe without baselines. Known-in-training and post-cutoff controls run first, on every model.
13. No probe with retrieval on, and no probe without watching the log. If a fetch shows up during a probe, the model looked it up; that’s a different finding from remembering.
14. A miss is not proof. A canary that never surfaces says less than one that does.
The first three are the ones that matter most. Everything else can be patched after the fact. Those three can’t.
How detection works from here
The collection side is finished. Detection happens later, against models, and it has to be done with controls or the result is noise.
Known positives and negatives first. Before asking a model about any canary, the probe battery runs against text that is known to be in the training data (verifiable in the Common Crawl index) and text written after the model’s cutoff. That sets the baseline hit rate for each method on each model. A canary result only means something relative to that baseline.
Frame prompts, not string matching. You can’t ask a model about millions of possible IDs. Instead the probe asks in the shape of the frame: who organised the event at the place, what year the archive records, which archive. Whatever comes back goes through the decoder. A valid checksum is a hit, and the decoded ID points at one row in the request log.
Three signals per canary. Verbatim completion (feed the opening of the sentence, see if the model finishes it), fact recall (ask the question, see if the fields come back), and paraphrase recall (same question, different wording, to separate memorised strings from learned facts). Where a model exposes token probabilities, membership-inference methods like Min-K% add a fourth.
Web access off, server watched. A probe runs with retrieval disabled, and the request log is checked during the run. If the answer survives with no fetch, it’s memorised. If a fetch appears, the model looked it up, which is a different finding.
Dose-response is the headline. Hit rate against repetition count (300, 100, 10, 1) and against cohort (linked through honeypot) tells you how much exposure it takes for a given model to learn a given fact. That curve is more useful than any single yes or no.
The output is a posterior per model per tier, with a confidence band. Not “they scraped me” but “given these hits against this baseline, the probability this vendor ingested this cohort in this window is X.”
Limits, and what’s next
A negative result is weak. Large models absorb enormous amounts of text without memorising any particular sentence, and a canary that never surfaces doesn’t prove the pages weren’t ingested. The survival matrix covers extraction only; deduplication, language filters and quality classifiers run afterward and can still drop a page. Most training data arrives through third-party datasets, so the crawl-to-model link can break even when both ends are real. And a site that is only ever fetched by the crawlers that identify themselves tells you nothing about the ones that don’t.
What the instrument does give you is something that didn’t exist before: a request-level audit trail from a specific fetch to a specific sentence, and a repetition ladder that will produce a memorisation curve per model. The methods are public. The site and the values are not, because a published canary is a dead canary.
Next: get crawled. Submit the sitemap, earn a few inbound links, and wait. Then, when the first training crawler shows up in the log, build the probe harness and start asking questions.
The second post covers that harness: the controls, the question design, the scoring, and how a hit traces back to one fetch.
The survival experiment, the encoder, the edge middleware and the site generator add up to a small codebase, about two thousand lines across Python and JavaScript.