Verified Agents
The problem with a User-Agent
Section titled “The problem with a User-Agent”Anyone can claim to be GPTBot. It is one header, and it costs nothing to copy.
That matters more than it used to. Sites increasingly let named AI crawlers through (to stay in search results, to be cited by assistants), and every one of those allowances is a hole if the claim is unverifiable. A scraper that sets User-Agent: GPTBot inherits whatever you granted OpenAI.
WebDecoy verifies the claim instead of trusting it.
For the trust model and product capabilities, see the Agent Identity product overview.
How an agent is verified
Section titled “How an agent is verified”Three methods, strongest first. Each detection records which one produced its verdict.
Signature (strongest)
Section titled “Signature (strongest)”Web Bot Auth is the IETF standard emerging from the webbotauth working group, with Cloudflare, Akamai, Amazon and OpenAI participating. The agent signs each request (RFC 9421 HTTP Message Signatures, Ed25519) and publishes its public keys at a well-known directory. We verify the signature against those keys.
This is proof, not provenance: the signature cannot be replayed onto another request, and an impersonator without the private key cannot produce one.
IP range and reverse DNS
Section titled “IP range and reverse DNS”For crawlers that don’t sign yet (most of them), we verify against the operator’s published IP ranges, refreshed automatically, and forward-confirmed reverse DNS: the PTR record must resolve back to an address in the operator’s own space. A PTR alone proves nothing; anyone can set one.
Ranges are tracked for Googlebot, Google-Extended, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, DuckDuckBot, Applebot and Applebot-Extended.
Claimed (unverified)
Section titled “Claimed (unverified)”The agent says who it is and nothing corroborates it. Not automatically hostile (plenty of legitimate tools have no verification story), but it earns no allowance.
Agent impersonation
Section titled “Agent impersonation”When a request claims to be a verified-capable agent and the proof fails (a signature that doesn’t verify, or an IP outside the operator’s published ranges), that is not merely “unverified”. Something is actively lying about its identity.
WebDecoy scores it as agent impersonation, a tripwire-grade signal that floors the threat score and marks the actor spoofed. It surfaces in detections with the claimed agent name, so you can see who is being impersonated against you.
The agent trust report
Section titled “The agent trust report”AI Crawlers in the sidebar opens AI Crawlers & Agents at /verified-agents. It is three sub-pages over one time window: Overview, Agents, and Handshakes. The older /ai-bots page folded into this report and now redirects here.
The window selector offers 24h, 7d and 30d, rides in the URL as ?period=, and survives switching between the tabs, so you can carry one window across all three views.
Overview
Section titled “Overview”The headline is every request that claimed a known agent, divided by what the evidence showed:
| Tile | What it counts |
|---|---|
| Requests claiming a known agent | the whole population the other three divide up |
| Proved who they are | claims backed by a signature, or by a published-IP and reverse-DNS match |
| Unproven claims | claimed, with nothing corroborating it either way |
| Forged identities | claimed, and failed the proof |
These are request counts, not agent counts. “Three verified crawlers” hides whether that was thirty requests or thirty thousand, and the request numbers are the ones worth quoting.
If anything was forged in the window, a callout above the tiles says how many impersonation attempts there were and links straight to those detections. Below the tiles, a volume ranking answers the other question the page exists for: who is taking the most.
Agents
Section titled “Agents”One row per agent, ordered by how much it claimed. Each row carries the agent’s category, the strongest proof that agent ever produced (a Web Bot Auth signature, an IP-range and reverse-DNS match, or nothing beyond the claim itself), and then the request counts behind it: how many claimed that identity, and how many of those were proven, unproven, or forged.
The last three always sum to the claimed total, because a request is exactly one of them. A non-zero forged count is a link into the detections that produced it, already filtered to that agent.
Strongest proof describes the agent’s best showing, not that row’s traffic. An agent can carry a verified badge and show forged hits, and that combination is the one worth looking at: genuine crawler traffic and impersonation of it arrived in the same window.
If every row shows an unproven claim and nothing else, look for the warning above the table. It distinguishes “no verifiable agent traffic yet” from “we cannot fetch the published IP ranges right now”, which are very different problems.
Handshakes
Section titled “Handshakes”The third tab reports the evidence a client cannot edit. Every client offers a list of TLS ciphers before it sends a single header, that list identifies the software, and whoever wrote the User-Agent has no say in it. Requests that claimed to be a browser while speaking TLS no browser speaks are counted and listed, with the number of distinct addresses behind each.
Only handshakes we have probed against a known client appear here. Anything else is left unclassified rather than guessed, and no enforcement decision is made from a handshake class.
Enforcement
Section titled “Enforcement”Verified agents can be allowed through the clearance gate without a token, by category (search engines, AI crawlers, monitoring). The toggles are on the Enforcement page’s Policy tab (/enforcement/policy), under Tokenless clients. Because the allowance keys on verification rather than the User-Agent string, turning on “allow AI crawlers” no longer means trusting a header.
An agent whose signature fails verification is denied that bypass and handled like any other untrusted client.
In your own application
Section titled “In your own application”The Node SDK performs the same signature verification locally, so you can make the decision in your own code rather than at the edge. See detectBot() and the webBotAuth() rule.
A note on where this is going
Section titled “A note on where this is going”The standard is pre-1.0 and moving. WebDecoy tracks the working group’s drafts and updates its implementation as they revise; the verification library is deliberately isolated so protocol churn touches one place. Signature coverage will improve as more operators adopt signing. Until then, IP-range and reverse-DNS verification carry most crawlers.
Related
Section titled “Related”- Clearance Enforcement: letting verified agents through the gate
- AI Scraper Scoring: how AI traffic is scored