Crawled, Indexed, or Cited: How to Read AI Traffic in Your Server Logs

If you run a website, three completely different things are happening to it right now, and your analytics package shows you none of them.

An AI company might be collecting your pages to train a model. It might be indexing you so its answer engine can find you later. Or someone may have asked a chatbot a question thirty seconds ago and it went to read your article in order to answer.

Those are three different relationships with three different implications, and most of the discussion about “AI traffic” mashes them into one. They are distinguishable — but only from raw server logs, and only if you know what to look at. The signals are not obvious, several of them are actively misleading, and the single most important one is a lie roughly half the time.

I relaunched this site in March 2026. What follows is five months of logs — 1,252,023 requests through 10 August — and the method for reading them. The numbers are mine; the method is the part worth stealing.

First: three jobs, three user agents

The major AI companies run separate crawlers for separate purposes and tell you which is which. This is the foundation of everything else:

That third row is the one people miss, and it is the interesting one. ChatGPT-User does not crawl. It fires when a person is sitting in front of a chat window, waiting, and the model has decided your page is worth reading to answer them. It is the closest thing to a reader in the entire log file.

Blocking is not one decision, either. You can refuse training and permit live answering, which is what most people actually want when they say they want to “block AI” — stay findable, stop feeding the corpus. Those are different user agents, so it is an available choice.

Then: assume the user agent is lying

Here is where it stops being simple. A user agent is a string the client chooses. Anyone can send any string. And AI crawler user agents are attractive to impersonate precisely because sites whitelist them.

Three of the four vendors above publish the IP ranges their crawlers use, as JSON, specifically so you can check:

• openai.com/gptbot.json, chatgpt-user.json, searchbot.json
• perplexity.ai/perplexitybot.json, perplexity-user.json
• developers.google.com/search/apis/ipranges/ — three files covering crawlers, special crawlers, and user-triggered fetchers

Others support the older method: reverse DNS. Look up the IP, confirm the hostname ends in the vendor’s domain, then forward-resolve that hostname back to the same IP. Amazon uses .crawl.amazonbot.amazon, Apple .applebot.apple.com, Microsoft .search.msn.com.

I checked every AI-crawler request in the window against whichever method its vendor offers. Verification here is weighted by requests, not by distinct IPs — one legitimate address doing ten thousand fetches should not be outvoted by ten impostors doing one each.

Read that column again. Only 12% of traffic claiming to be Googlebot actually was. Only 41% of live ChatGPT fetches. Two percent of Meta’s crawler.

If you have ever built a dashboard of “AI crawler traffic” straight from user-agent strings — and that is what nearly every published analysis of this does, including my own first pass — the number you got was wrong by a factor of two to eight, in the direction that makes the story more dramatic.

The proof, in case the percentages don’t land

Here is what the traffic claiming to be Googlebot was actually doing:

151  /.git/config
144  /.env.production
141  /.gitlab-ci.yml
130  /.env.local
130  /.env
130  /.github/workflows/deploy.yml

8,465 requests hunting for credentials, wearing Googlebot’s name. Meta’s crawler string was used for 2,705 such probes — more than a third of all traffic claiming that identity. Every single AI crawler user agent in my logs was used at some point to request /.env or /credentials.json, including GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot.

Scanners impersonate these crawlers deliberately. A site that whitelists Googlebot to be helpful has whitelisted whoever is willing to type the word Googlebot.

The unverified traffic also tends to come from somewhere revealing. Unverified ChatGPT-User requests mostly resolve to Google Cloud VMs and budget VPS hosts. Roughly 70% of the traffic claiming to be Meta’s crawler came from Google Cloud address space. Real crawlers run from their own infrastructure.

An important fairness caveat

Two entries in that table show “—” rather than a percentage, and the distinction matters.

Anthropic publishes no crawler IP ranges and sets no reverse DNS. There is no method available, so ClaudeBot cannot be verified by anyone. That is not the same as failing verification, and I want to be precise about it because the sloppy version of this analysis would report ClaudeBot as “0% verified” and imply it is fake.

The behavioural evidence says otherwise: 91% of ClaudeBot traffic came from two adjacent blocks, 216.73.216.0/24 and 216.73.217.0/24. That concentration is what a single legitimate operator looks like. Only 145 of its 3,688 requests were credential probes. It is almost certainly the real thing — I simply cannot prove it, and neither can you.

That is a gap worth naming. Publishing an IP range is a small piece of work that makes a crawler accountable and makes impersonating it pointless. OpenAI, Perplexity and Google have done it. Anthropic and Meta have not, and the consequence lands on site operators.

One more caveat on method: reverse DNS numbers above are measured across each crawler’s top 200 addresses, which covers 38–100% of that crawler’s requests depending on how concentrated it is. Published-range figures cover everything.

Now the actual question: are you being used as a source?

Being crawled is not being cited. A fetch means your page was read while an answer was composed. Whether you were then named as a source, or quietly absorbed, is not in your logs. There is no field for it.

But there is one clean signal, and almost nobody knows about it.

When ChatGPT surfaces a link, it appends ?utm_source=chatgpt.com to the URL. Perplexity appends utm_source=perplexity. So when someone clicks a citation, that tag arrives in the request path of the landing hit. It is proof of two things at once: you were shown as a source, and a human thought it was worth clicking.

It also survives where the referrer does not. Of the five ChatGPT citation clicks I can identify by utm tag, only two carried a matching referrer — the rest were stripped by privacy settings or in-app browsers. If you count AI referrals the usual way, you miss more than half of them.

Five months, measured both ways:

Set that against the crawling. There were 1,409 verified ChatGPT live fetches in the window and five tagged citation clicks — about one click per 280 times a model read a page to answer somebody. Counting every method together gives roughly 26 human arrivals from AI products, against something like 22,000 verified AI crawler requests. Call it 850 to 1.

A trap I fell into on the first pass, so you don’t have to: do not grep whole log lines for chatgpt.com. The tag persists in the referrer as a visitor clicks around your site, so internal navigation looks like a stream of new arrivals. I turned six clicks into thirty-five that way. Match the request path for the utm tag, and the referrer field for the host — never the raw line.

Training and answering want different things

This is the finding I did not expect, and it is the one with practical consequences.

Compare what each crawler went for. GPTBot, collecting training data, spread 9,027 requests across 928 distinct URLs with no page taking more than 1% — and its most-fetched articles were my technical writing on agent architecture, security and design patterns.

ChatGPT-User, answering live questions, did the opposite: 49% of its requests went to a single article, a comparison piece full of benchmark tables. Claude-User behaved identically — 50% on the same post.

Training crawlers want breadth and depth: your durable, structured, technical material, all of it. Live fetchers want the specific thing that answers a recurring question, over and over, and that turns out to be comparison content — dense, tabular, factual-looking, exactly what an answer engine cannot generate for itself.

That split is a useful lens on your own archive. The writing I consider most durable is what gets absorbed into models. The writing I had written off as perishable is what makes me a live source. Those are different jobs, and it is worth knowing which of your pages is doing which.

The breadth-versus-depth pattern is also a verification signal in its own right. A training crawler spreading evenly across hundreds of URLs from a hundred addresses looks nothing like a live fetcher hammering one article from a thousand addresses. When the behaviour doesn’t match the claimed job, be suspicious.

Do it on your own logs

Everything above came from grep, awk, and about forty lines of Python. Three commands get you most of the way.

What is claiming to visit you, split by job:

zcat -f /var/log/nginx/access.log* | grep -ohiE \
 "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|\
Perplexity-User|Google-Extended|meta-externalagent|Bytespider|Amazonbot|CCBot" \
 | tr 'A-Z' 'a-z' | sort | uniq -c | sort -rn

Whether you are being cited — note this reads the request path, not the whole line:

zcat -f /var/log/nginx/access.log* \
 | awk -F'"' '{split($2,a," "); if (a[2] ~ /utm_source=/) print a[2]}' \
 | grep -oE 'utm_source=[^&" ]*' | sort | uniq -c | sort -rn

Whether any of it is real. Fetch the vendor’s published ranges and check membership — this is the step that changes your numbers:

curl -s https://openai.com/chatgpt-user.json | grep -o '"ipv4Prefix": "[^"]*"'
# then test each logged IP for membership, e.g. with python3 -c ... ipaddress

For vendors without published ranges, reverse DNS, and confirm the hostname forward-resolves back to the same address — a PTR record alone can be set by whoever controls the IP.

What I take from this

I am not blocking anyone. Verified AI crawling is under 2% of my traffic; the dominant load on this server is people trying to brute-force wp-login.php, which accounted for 38% of all requests in the window and is a far more boring story. And I would rather be in the corpus than out of it.

But three things changed in how I measure this site.

User-agent counts are marketing, not measurement. Any figure you have seen about AI crawler volume that wasn’t IP-verified is inflated, mine included, and the inflation isn’t uniform — it’s worst exactly where the story is most interesting.

The utm tag is the only honest citation signal available. It is undercounted by everyone using referrer analytics, and it is the one number that tells you whether an answer engine is sending real people to you rather than just reading you.

Fetches without clicks are still reach. A verified live fetch means a human asked a question and my page was part of the answer. Fifty of those produce one visitor. Counting only the visitor throws away the other forty-nine, and the gap between what analytics shows and what is actually happening is going to keep widening.

None of this required a tool, a service, or a tracking script. It was sitting in /var/log/nginx/ the whole time.