Building the detector: how to ask a model whether it ate your website
The first post covered the bait side: a site built to be scraped, seeded with traceable canaries, fronted by an edge layer that stamps every bot with a unique sentence. This post covers the other half. Once the pages are out there, how do you find out whether a model learned them?
The short answer is that you ask, carefully, with controls, and you never trust a single answer.
The detector is a calibrated interrogation, not a search
Most “did AI train on my site” tools do one thing: paste a sentence into a chatbot and see if it looks familiar. That has two problems. The chatbot may be retrieving your page live rather than remembering it, and you have no idea what the answer would look like if the model had never seen your site at all.
So the detector has three target populations, probed identically:
| Population | What it is | What it measures |
|---|---|---|
| Canaries | The sentences actually planted on the site, by tier and repetition count | The signal |
| Negatives | Canaries generated from a second seed and never published anywhere | The false-positive floor |
| Positives | Well-known historical facts, phrased in the same sentence shapes | The recall ceiling |
A canary hit only means something relative to those two baselines. If a model “recalls” never-published negatives 5% of the time, a 5% hit rate on real canaries is nothing. If it recalls famous facts only 60% of the time with these question shapes, a 60% hit rate on canaries is everything.
Three ways to ask
Each target gets three kinds of question, because memorised strings and learned facts behave differently.
Completion. Give the model the opening of the sentence, cut just before the person’s name, and ask it to continue. This catches verbatim memorisation. It is the probe that mattered in the newspaper lawsuits.
Recall. Ask a direct question that supplies some fields and asks for the rest: who is credited with the event at the place in that year; what year the archive gives; which record book documents the person. Three variants, each withholding a different field.
Paraphrase. Ask about the person and place with no reference to the sentence structure at all. A model that answers this learned the fact, not the string.
Every probe runs at temperature 0 and again at a higher temperature with repeats, because a memory the model is unsure of surfaces more often when sampling is loosened.
The prompts never contain the full canary sentence. A completion prefix stops before the identifying fields; a recall question supplies only what it has to. The system prompt forbids browsing, and retrieval is never requested from any API, but neither of those is the real guard. The real guard is the edge log: the harness records the timestamp when a run starts and counts fetches of the bait site during the run. A fetch means a hit might be retrieval, not memory, and the report says so.
Scoring an answer
Every answer passes through the same decoder that reads canaries out of extracted web pages. Two things come back:
- Field hits. Which of the six fields (given name, surname, place, event, archive, year) appear in the answer, matched as whole words. A given name alone is worth little: one chance in 4,096. Four fields together are beyond coincidence.
- Full decode. Whether the decoder reassembled the exact 48-bit ID, with the checksum word and year validating. This is the forensic result: the ID points at one manifest entry or one row in the request log.
A target counts as hit when any answer decodes fully, or when at least two separate answers return four or more fields. The second condition exists because high-temperature sampling will occasionally produce a lucky guess, and one lucky guess shouldn’t move the needle.
From hits to probabilities
Hits are grouped by tier (site, cohort, repetition ladder, vendor, window) and by how many times the canary appeared on the site. For each group:
- Hit rate with a Wilson 95% interval, because groups are small and a plain proportion lies.
- Excess over the negative floor. Subtract the rate at which the model “recalled” the never-published controls.
- Normalised by the positive ceiling. Divide by the rate at which it recalled famous facts with the same questions, so a model that is bad at this question shape isn’t penalised.
- Posterior. Start from a prior taken from the crawl log (did verified training crawlers from this vendor fetch these pages at all?), multiply by the likelihood ratio implied by the two control rates, and read off the probability that the group was ingested.
The posterior is honest for the high-repetition tiers. For canaries that appeared once, it is pessimistic by construction: famous facts set a recall ceiling that no once-seen sentence will reach. For those, the finding is the curve, not the number.
The headline chart is a dose-response curve
Hit rate on the vertical axis, occurrences on the site on the horizontal, log scale, one point per rung of the repetition ladder (300, 100, 10, 1), with the negative floor and positive ceiling drawn as reference lines. That curve says how much exposure it takes for a given model to learn a given fact. Published memorisation studies estimate this with synthetic insertions into training runs they control; this measures it from the outside, on a production model, with a site the model’s crawler actually visited.
A second chart worth drawing once there’s data: hit rate by cohort. If the linked pages score and the sitemap-only pages don’t, the crawler followed links and ignored the sitemap. If the honeypot cohort scores, the crawler ignored robots.txt and followed an invisible link, and the request log has the row to prove it.
Membership inference, where the API allows it
For models that expose per-token log-probabilities (open-weight models served locally, some commercial endpoints), the harness also computes Min-K% Prob on the full canary sentence: the mean log-probability of the least likely 20% of tokens. Text a model has seen tends to have fewer surprising tokens. The threshold comes from the negative controls, 95th percentile, so the method is calibrated the same way as everything else. It is noisy on large models and useless without logprobs, which is why it’s a fourth signal rather than the first.
What a run looks like
A run against one model: a few dozen canary targets, twenty negatives, eight positives, three question types, two temperatures, two repeats. About 1,500 short API calls, ten to fifteen minutes, a few dollars. The output is a table per tier and a JSON file of hits by canary ID. Nothing in storage is the canary text itself; IDs regenerate it from the seed when needed.
The first runs, taken while the site is days old, should show every tier at the negative floor. That’s not a failure. That’s the baseline every later run is compared to, and the day a tier lifts off that floor is the day the experiment pays out.
What you have to keep secret
The method is public by design. Five things are not, and the experiment depends on all five staying that way:
- The site’s identity.
- The seed that generates the content and static canaries, and the secret that derives the edge tiers.
- Any rendered canary sentence, in any post, chat, or probe transcript shared outside the harness.
- The vocabulary lists. The construction can be described; the lists let anyone scan their own corpus for the canaries.
- The control seed, so the negatives stay unpublished.
One more, subtler than the rest: this post will itself be crawled and trained on. The sentence shapes described here are deliberately not the live ones, and no example in either post carries all six fields. If a future model completes a probe from the blog instead of the bait, the negative controls are built to catch it. Better not to need them.
Limits
Negative results are weak; absence of recall is not absence of ingestion. Commercial models can filter or down-weight content, and a vendor who reads this could filter sentences in these shapes, which would show up as a null result sitting on top of a log full of their verified fetches. Positive controls calibrate recall for well-known facts, not for the long tail where canaries live. Logprob methods only work where logprobs exist. And a probe can only find what it asks about, so a model that absorbed the site’s style without a single fact is invisible to it.
What the detector does deliver is the thing that was missing: a repeatable, controlled measurement, run on a fixed schedule against a known exposure, that either stays flat or doesn’t. When it doesn’t, the hit carries an ID, and the ID carries a fetch.
Code for the harness, the decoder and the edge layer is a small Python and JavaScript codebase. The vocabulary generation ships as “bring your own lists” for the reason above.