Verified Agents

The problem with a User-Agent

Anyone can claim to be GPTBot. It is one header, and it costs nothing to copy.

That matters more than it used to. Sites increasingly let named AI crawlers through — to stay in search results, to be cited by assistants — and every one of those allowances is a hole if the claim is unverifiable. A scraper that sets User-Agent: GPTBot inherits whatever you granted OpenAI.

WebDecoy verifies the claim instead of trusting it.

How an agent is verified

Three methods, strongest first. Each detection records which one produced its verdict.

Signature (strongest)

Web Bot Auth — the IETF standard emerging from the webbotauth working group, with Cloudflare, Akamai, Amazon and OpenAI participating. The agent signs each request (RFC 9421 HTTP Message Signatures, Ed25519) and publishes its public keys at a well-known directory. We verify the signature against those keys.

This is proof, not provenance: the signature cannot be replayed onto another request, and an impersonator without the private key cannot produce one.

IP range and reverse DNS

For crawlers that don’t sign yet — most of them — we verify against the operator’s published IP ranges, refreshed automatically, and forward-confirmed reverse DNS: the PTR record must resolve back to an address in the operator’s own space. A PTR alone proves nothing; anyone can set one.

Ranges are tracked for Googlebot, Google-Extended, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, DuckDuckBot, Applebot and Applebot-Extended.

Claimed (unverified)

The agent says who it is and nothing corroborates it. Not automatically hostile — plenty of legitimate tools have no verification story — but it earns no allowance.

Agent impersonation

When a request claims to be a verified-capable agent and the proof fails — a signature that doesn’t verify, or an IP outside the operator’s published ranges — that is not merely “unverified”. Something is actively lying about its identity.

WebDecoy scores it as agent impersonation, a tripwire-grade signal that floors the threat score and marks the actor spoofed. It surfaces in detections with the claimed agent name, so you can see who is being impersonated against you.

The Verified Agents dashboard

Verified Agents in the sidebar shows every agent seen on your properties, folded by name, with the strongest verification method each achieved and a count of impersonation attempts. It answers two questions directly: which AI crawlers are actually reading your site, and who is pretending to be them.

If everything reads claimed, check the health signal on that page — it distinguishes “no verifiable agent traffic yet” from “we cannot fetch the published IP ranges”, which are very different problems.

Enforcement

Verified agents can be allowed through the clearance gate without a token, by category — search engines, AI crawlers, monitoring — under Enforcement → Setup → step 4. Because the allowance keys on verification rather than the User-Agent string, turning on “allow AI crawlers” no longer means trusting a header.

An agent whose signature fails verification is denied that bypass and handled like any other untrusted client.

In your own application

The Node SDK performs the same signature verification locally, so you can make the decision in your own code rather than at the edge. See detectBot() and the webBotAuth() rule.

A note on where this is going

The standard is pre-1.0 and moving. WebDecoy tracks the working group’s drafts and updates its implementation as they revise; the verification library is deliberately isolated so protocol churn touches one place. Signature coverage will improve as more operators adopt signing — until then, IP-range and reverse-DNS verification carry most crawlers.

Clearance Enforcement — letting verified agents through the gate
AI Scraper Scoring — how AI traffic is scored