42 percent of our AI crawl was not AI. How to measure yours

Through August we were convinced that AI models had pulled content from seedlight.eu 6,600 times. The number looked healthy and it flattered our theory that this channel was working. Then we broke the logs down by path and found that 2,842 of those fetches were credential scanners wearing an AI crawler name in the header. That left 3,934. Here are the three mistakes that inflated the figure, and how to check yours.

Three numbers that did not mean what we thought

Measured on our own logs at seedlight.eu across a 30 day window closing 27 August 2026. The right-hand columns are the same raw figure once requests for paths that are never content are stripped out.

seedlight.eu, 29 July to 27 August 2026. "Real" means a path that is an actual page of the site; the remainder are requests for config files and system directories.

The last column is the interesting one. Of 348 fetches attributed to Anthropic, 296 returned something other than 200, meaning they hit an address that does not exist. A genuine crawler follows your sitemap and your links, so it almost always lands on a page that is there.

Bot Score reads 99 for everything

Cloudflare describes its bot score as a scale from 1 to 99, where 1 means almost certainly automated and 99 means almost certainly human. That reads like a filter you can lean on, and lean on it we did: anything scoring below 30 was binned as a bot.

The filter never once fired.

We pulled the distribution of stored values across 30 days and got a single answer:

• 4,516 server-side pageviews scoring 99
• 470 confirmed pageviews scoring 99
• Zero requests carrying any other value

Tor exit nodes scored 99. So did machines on Alibaba Cloud and the Ahrefs crawler. The reason sits in the documentation: granular bot scores go only to Enterprise customers who have bought Bot Management, while everyone else gets bot groupings in Bot Analytics. The field exists on the request object and looks like data, but it measures nothing.

That is worse than having no signal. A missing field announces itself the first time you chart it; a constant looks like a working filter and can keep looking like one for months.

Scanners borrow Amazonbot’s name

A crawler name lives in the User-Agent header, which anybody sets for themselves. Nothing verifies it, and nothing stops a scanner hunting for other people’s credentials from introducing itself as Amazon’s bot. These are the addresses our AI-labelled traffic was knocking on:

• /.env, /.env.backup, /app/.env, /api/.env
• /.git/HEAD and /.svn/entries
• /firebase-adminsdk.json and /env.json
• /@fs/proc/self/environ

No language model needs your database password.

These are scanner signatures working through a list of common configuration leaks, betting that somewhere a file of secrets went live alongside the rest of a site. The AI label was earned by claiming it, and by nobody checking the claim.

Strip those paths out and the ranking of who actually reads us changes beyond recognition. Amazon and Anthropic, the two largest entries in the raw table, turn out to be almost entirely impersonation. Our real reader is Perplexity: 724 fetches of addresses that exist, mostly answered with a 200.

GPTBot, OAI-SearchBot and ChatGPT-User are three different events

The third mistake costs the most, because it erases the only signal that anticipates anything. Model providers run separate bots for separate jobs, and they describe the difference themselves.

Descriptions per OpenAI and Perplexity documentation, accessed 1 September 2026.

Most analytics setups, ours until last week included, match a fragment of the name to a provider and stop there. All three OpenAI bots then land under one label, and with them goes the answer to whether anyone is asking about you at all.

One file we never saw

The last mistake looks harmless. Our middleware logged only responses with a text/html content type, because pageviews were what we cared about. A sensible assumption, right up until you ask a question about llms.txt.

That file is served as text/plain. Which means not one fetch of it across 60 days left any trace in our data. We wrote a file specifically for models and then cut ourselves off from the only evidence of whether any of them read it.

A second problem surfaced alongside it. Astro generates the sitemap as sitemap-index.xml, while bots were knocking on /sitemap.xml and /sitemap_index.xml and collecting 404 error pages. The address in robots.txt was correct, but not every fetcher reads robots first.

How to measure yours

None of this needs a tool or a subscription. It needs four columns in your logs and one principle: never trust what a request says about itself.

• Log the response status. Filtering to 200 removes most scanners without any list of names, since by definition they fire at addresses you do not have.
• Log the originating network. The operator name (asOrganization on Cloudflare) is harder evidence than a User-Agent: impersonating Chrome is free, renting a consumer line is not.
• Store the full bot name, not just the provider. Otherwise you lose the difference between training and a question asked live.
• Log responses that are not HTML. Without that, llms.txt, robots.txt and your sitemap are invisible to you.

Then put two things side by side: who reads you, and whether anything comes back. Across 60 days we recorded seven visits referred by AI assistants against thousands of fetches. That gap is normal and not bad news in itself, but without the numbers separated you cannot tell whether it is closing.

What these numbers cannot tell you

This is one site, and a small young one at that. Your proportions will differ, particularly with an older domain or a catalog running to thousands of product pages. The mechanism repeats; the specific 42 percent does not.

Two further caveats. Filtering by network name is a heuristic, so alongside the scanners it also catches real traffic from corporate lines, VPNs and company networks that happen to sit in a cloud; that error runs the opposite way to the earlier ones, and it cost us some genuine visitors. Fetch counts say nothing about citations either, since a model can fetch a page and never use it, and can equally quote you from memory without fetching anything at all.

If you would rather start with the store than the logs, first check whether an agent has anything to buy from you at all. Measuring traffic makes sense once there is a reason to arrive. For the wider question of how you reach model answers in the first place, we covered it in our piece on GEO for eCommerce, and what models actually say about you is a separate metric, described in our piece on brand visibility.

At Seedlight we run this measurement in the Maintenance and Growth stage of the BEAM framework, alongside the product record, because either one alone says little. An honest caveat: none of this guarantees a model will start citing you, because nobody controls that. What you control is whether your own numbers tell the truth.

It is worth remembering that a growing share of visibility happens off your domain, where no log will ever show it. On a marketplace the platform’s own algorithm decides it and you measure it in a completely different way; that side is run by our sister agency Amazonway in marketplace account management.