Between March and August, 12,134 requests to this site announced themselves as Googlebot. 8,725 of them — 71.9% — were hunting for credentials. /.env, /.git/config, /.aws/credentials, deploy workflows, database dumps.
Google was not doing this. Not one of those requests came from an address Google publishes.
Somewhere along the way, “Googlebot” stopped being an identity and became a costume — and the same is now true of GPTBot, ClaudeBot, PerplexityBot and the rest. The reason is straightforward, and it is our fault rather than theirs: we told everyone these names get waved through.
The number that settles it
Across five months and 1,265,337 requests, 171,555 were credential probes. Most wore no disguise at all — ordinary scanner traffic, the background radiation of running anything on the public internet.
But 13,670 of those probes arrived wearing the user agent of a named AI or search crawler. So I checked every one against the vendor’s published IP ranges.
Zero came from a verified vendor address.
Not “a small number.” Zero. My first pass reported seven, and all seven turned out to be genuine Googlebot crawling spam search-result URLs on my own site that happened to contain the word “dumps” — my pattern matching, not Google’s behaviour. Once those came out, the count was clean.
That is the whole finding, and it is worth stating plainly because it makes the mitigation obvious: no legitimate AI crawler is probing your site for secrets. Every request that appears to be one is an impostor, and a single check separates them.
Which costumes get worn
The share of each identity’s traffic that was credential probing:
| Claimed identity | Probes | Share of its traffic |
|---|---|---|
| Googlebot | 8,725 | 71.9% |
| Google-Extended | 120 | 69.0% |
| Perplexity-User | 51 | 56.7% |
| meta-externalagent | 3,057 | 39.3% |
| CommonCrawl CCBot | 69 | 24.6% |
| PerplexityBot | 176 | 20.2% |
| OAI-SearchBot | 185 | 15.9% |
| ChatGPT-User | 359 | 10.2% |
| ClaudeBot | 157 | 4.1% |
| Amazonbot | 314 | 4.3% |
| GPTBot | 174 | 1.9% |
| Bytespider | 41 | 0.5% |
| bingbot | 55 | 0.4% |
Read that as a ranking of which names are worth stealing. Googlebot tops it by a distance, because Googlebot is the identity every webmaster has been trained for twenty years not to block. Google-Extended — the opt-out control for AI training — sits second, which is a bleak little detail: the user agent that exists so publishers can decline is itself being impersonated to break in.
At the bottom sit GPTBot, Bytespider and bingbot at under 2%. Those are crawlers behaving like crawlers.
What they were after
The top targets, in order:
224 /.git/config
221 /.env.production
215 /.env.local
211 /.env
204 /.gitlab-ci.yml
191 /.github/workflows/deploy.yml
146 /.aliyun/credentials
133 /.tencent/credentials
129 /.qcloud/credentials
128 /.aliyun/config.json
127 /.env.staging
97 /.env.bak
95 /.env.backup
87 /secrets.json
74 /.aws/credentials
72 /credentials.jsonThe .env family and exposed .git directories are the classics — a committed .env or a web-served .git hands over database passwords and API keys in one request.
The CI files are the more modern touch. /.gitlab-ci.yml and /.github/workflows/deploy.yml rarely contain secrets directly, but they name your deployment targets, your registries, and the secret names your pipeline injects. That is reconnaissance rather than a smash-and-grab.
And note the Alibaba, Tencent and Qcloud credential paths sitting alongside AWS. Those appear in fixed proportion with everything else, which suggests a single toolkit with a broad wordlist rather than a targeted operation. Nobody looked at this site and decided it probably ran on Qcloud.
Where it actually comes from
Here is the detail I keep returning to. The three busiest sources of probes wearing Meta’s crawler identity:
1,047 34.63.249.85 → 85.249.63.34.bc.googleusercontent.com
715 34.91.139.167 → 167.139.91.34.bc.googleusercontent.com
667 34.14.90.240 → 240.90.14.34.bc.googleusercontent.comMeta’s crawler, running on Google Cloud. The requests wearing OpenAI’s ChatGPT-User came from the same place. This is rented compute with a forged label, and the reverse DNS gives it away in one lookup.
The single busiest impostor — 6,092 probes as Googlebot from 101.47.155.116 — has no reverse DNS at all. Real Googlebot resolves to crawl-*.googlebot.com. Every time. The absence of a PTR record is not proof of much on its own, but combined with “claims to be Googlebot” it is conclusive.
It is not even a disguise. It is a rotation.
That address is worth a closer look, because it reframes the whole thing. Its user agents across the campaign:
3,261 curl/7.68.0
3,220 python-requests/2.28.0
3,138 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ...
3,125 Mozilla/5.0 (X11; Linux x86_64) ... Chrome/120
3,111 Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) ...
3,072 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)Six identities, near-identical counts. Googlebot is not a targeted disguise — it is entry six on a list, cycled per request. Here are six consecutive requests from that address:
01:40:54 /.git/config Googlebot/2.1
01:40:55 /oss.yaml Firefox 121 on Windows
01:40:56 /.env. Chrome on Windows
01:40:56 /credentials.js python-requests/2.28.0
01:40:57 /.env.local%00 (no user agent)
01:40:57 /.env.local~ Chrome on WindowsFive identities in three seconds, from one address.
If your rules challenge curl, wave through Googlebot, and rate-limit by user agent, this scanner is simultaneously blocked, allowed, and counted as five separate clients — while remaining one machine working through a wordlist. Any control keyed to the user agent is not making a decision about the client. It is making a decision about a string the client rewrites between requests.
This is recent, and it arrived all at once
Probes wearing a crawler costume, by month:
| Month | Probes |
|---|---|
| March | 23 |
| April | 45 |
| May | 85 |
| June | 191 |
| July | 9,396 |
| August (to the 10th) | 3,930 |
A forty-nine-fold jump in a month. And it was not gradual: 6,451 of July’s probes landed on a single day, 24 July. One campaign, one afternoon, and then a sustained background level roughly twenty times higher than before it.
Whatever tooling did this picked up the technique between June and July. If you have not looked at your own logs since the spring, you are looking at a different threat than the one you remember.
Why this breaks the standard advice
The conventional guidance for bot management is to allowlist the good crawlers so you do not accidentally block search traffic, then rate-limit or challenge the rest. Practically every WAF ships a “verified bots” or “known good bots” toggle built on this idea.
The trouble is that a large amount of that tooling matches on the user agent string, which is a value the client chooses. If your allowlist rule is “user agent contains Googlebot, skip the challenge,” you have not built an allowlist. You have published a password, and it is in every scanner’s default configuration.
This is the same structural mistake that shows up inside agent systems: trusting a self-reported identity because it is inconvenient to verify. In agent design it is the confused deputy — a component with real authority acting on instructions from something that merely claimed the right to give them. Here the deputy is your firewall.
What to do about it
Verify, or do not allowlist. There is no middle option that means anything. Verification is not hard — three of the major vendors publish their ranges as JSON you can fetch and cache:
openai.com/gptbot.json · chatgpt-user.json · searchbot.json
perplexity.ai/perplexitybot.json · perplexity-user.json
developers.google.com/search/apis/ipranges/*.jsonFor the rest, use reverse DNS and then forward-resolve the hostname back to the same address. A PTR record alone can be set by whoever controls the IP; the round trip is what makes it evidence. Amazon uses .crawl.amazonbot.amazon, Apple .applebot.apple.com, Microsoft .search.msn.com.
Alert on the combination, not the paths. Credential probing is constant and mostly noise. “Credential probe plus a crawler user agent” is a much rarer signal, and on this evidence it is never legitimate. That makes it a clean, low-volume alert with essentially no false positives — the best kind.
Check what an allowlisted identity can actually reach. If skipping the bot challenge also skips rate limiting, one forged header buys an attacker unlimited attempts. Whatever the allowlist grants is what impersonating it is worth.
Then fix the boring thing. All of this only matters because a .env file might be sitting in a web root somewhere. The scanning is unavoidable; what it finds is not.
The general shape of it
A trusted identity that anybody can assert, checked by nobody, granted real privileges. We have seen this before with email senders, and it took SPF, DKIM and DMARC to make sender identity mean anything.
Crawler identity is at the pre-SPF stage right now. The verification mechanisms exist and some vendors publish what you need — but the checking is optional, so mostly it does not happen, and the gap between “claims to be Googlebot” and “is Googlebot” is where 13,670 requests went looking for my secrets.
Your logs already contain your version of this. It takes about a minute to find out.
Method, and why the user agent is unreliable in the first place: Crawled, Indexed, or Cited. On self-reported identity as an architectural problem inside agent systems: Threat Modeling an AI Agent.
