AI Scraper Scoring

WebDecoy includes a dedicated scoring dimension for detecting AI training crawlers. This runs completely independently from attack detection because the signal interpretation is fundamentally different.

The Problem: AI Scrapers Don’t Look Like Attackers

When GPTBot crawls your website, it honestly identifies itself in the User-Agent header. From an attack detection perspective, this is low threat: the bot isn’t trying to hide or evade detection.

But from a content protection perspective, this is the highest possible threat: a known AI training crawler is actively collecting your content.

Same signal, opposite interpretations:

┌─────────────────────────────────────────────────────────────────┐
│                        GPTBot Request                            │
│   User-Agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like       │
│   Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)    │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│   Attack Detection View          │   Content Protection View     │
│   ──────────────────────         │   ────────────────────────    │
│   "This bot isn't hiding"        │   "Known AI training crawler" │
│   User-Agent easily spoofed      │   Confirmed identity          │
│   Score: 14 (MINIMAL)            │   Score: 85+ (CRITICAL)       │
│   Verdict: Low threat            │   Verdict: High threat        │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Why Two Separate Scores?

WebDecoy shows both scores side-by-side because they answer different questions:

Score	Question It Answers
Attack Risk	“Is this visitor trying to attack or evade detection?”
AI Scraper Risk	“Is this visitor an AI crawler collecting content for training?”

For a detected GPTBot request:

┌─────────────────────────┬─────────────────────────┐
│      Attack Risk        │    AI Scraper Risk      │
│                         │                         │
│          14             │           85+           │
│       MINIMAL           │        CRITICAL         │
│                         │                         │
│  "Not trying to hide"   │   "GPTBot detected"     │
└─────────────────────────┴─────────────────────────┘

Key insight: Honest self-identification is LOW threat for attack detection but HIGH confidence for AI scraper detection.

When the identity is not merely honest but verified, the split widens by design: the attack score is capped at 60 absent a hostile act, while the AI scraper score is unchanged. See What verification does to each score.

Publisher Value

For publishers and content creators, AI Scraper scores provide the visibility that attack scores alone cannot deliver. This is the core value proposition for content protection.

---

How AI Scraper Scoring Works

Inverted Signal Weighting

AI Scraper scoring uses inverted signal weighting compared to attack detection:

Signal	Attack Scoring	AI Scraper Scoring	Reasoning
Honest bot UA (GPTBot, ClaudeBot)	1% weight (trivially spoofed)	Registry base score, 85 for these	For attack detection, UA can be spoofed. For AI detection, honest self-ID is reliable
Datacenter IP	3% weight (VPN false positives)	+10 points boost	AI crawlers typically operate from cloud infrastructure
Missing Referer	1% weight (easily spoofed)	+5 points boost	Direct access patterns are typical for crawlers

Both dimensions read the same registry

The crawler tables below are not a second list maintained beside the attack scorer’s. Both dimensions ask the shared agent registry what a client is, and take its answer.

That was not always true on the attack side. The attack scorer used to derive its User-Agent component from a substring rule: any User-Agent containing bot scored 100. That rule is blind in both directions. ChatGPT-User matched on +https://openai.com/bot, the courtesy URL a crawler publishes precisely so it can be identified, and scored 100. ByteSpider contains no bot at all, fell through to a generic suspicious-UA rule, and scored 35 against a registry base of 75, despite being the only agent in that comparison that ignores robots.txt.

Both dimensions now sit in the order the registry assigns. If you filter your dashboard on a category name, write a bots() rule against it in the Node SDK, and read an attack score for the same request, all three are describing the same table.

Detection Process

┌─────────────────────────────────────────────────────────────────┐
│                 AI SCRAPER DETECTION PIPELINE                    │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│   Incoming Request                                               │
│         │                                                        │
│         ▼                                                        │
│   ┌───────────────────────────────────────────┐                 │
│   │     CHECK USER-AGENT PATTERNS             │                 │
│   │  GPTBot? ClaudeBot? CCBot? ByteSpider?    │                 │
│   └─────────────────┬─────────────────────────┘                 │
│                     │                                            │
│        ┌────────────┴────────────┐                              │
│        │                         │                               │
│        ▼                         ▼                               │
│   ┌─────────┐              ┌───────────┐                        │
│   │ MATCHED │              │ NO MATCH  │                        │
│   └────┬────┘              └─────┬─────┘                        │
│        │                         │                               │
│        ▼                         ▼                               │
│   ┌───────────────┐        ┌───────────────┐                    │
│   │ Base Score    │        │ Score 0       │                    │
│   │ + Confidence  │        │ Category      │                    │
│   │ + Category    │        │ "none"        │                    │
│   └───────┬───────┘        └───────┬───────┘                    │
│           │                        │                             │
│           └──────────┬─────────────┘                             │
│                      ▼                                           │
│            ┌─────────────────┐                                   │
│            │  AI Scraper     │                                   │
│            │  Score + Level  │                                   │
│            │  + Category     │                                   │
│            └─────────────────┘                                   │
└─────────────────────────────────────────────────────────────────┘

Scoring Components

The AI Scraper Score exists only for a User-Agent that matches an agent in the registry. It starts at that agent’s base score, and two supporting signals can raise it (capped at 100):

Signal	Points
Matched agent	The agent’s registry base score (training crawlers 65-85, search crawlers 30-40)
Datacenter/hosting IP	+10
Missing Referer	+5

Confidence is set by the agent’s category, from 0.95 for training crawlers down to 0.80. A User-Agent that matches no agent scores 0 with category none; behavior never produces an AI scraper score.

---

What verification does to each score

A crawler’s declared identity can be checked: against the IP ranges the operator publishes, against a forward-confirmed reverse DNS record, or against a Web Bot Auth signature. The outcome of that check pulls the two dimensions in opposite directions, which is the clearest illustration there is of why they are separate numbers.

Verification outcome	AI Scraper Risk	Attack Risk
Verified	Unchanged. It really is GPTBot, and it really is reading your content	Capped at 60, the top of MEDIUM, absent a hostile act
Unverified	Unchanged. The claim is still worth logging	Scored on its own merits: the worse of the registry base and the generic heuristic
Spoofed	The name is reported with the claim marked as failed	Hits the impersonation floor

A verified agent is capped at the top of MEDIUM

The attack dimension measures how automated and how evasive a client looks. A proven identity answers both questions at once: a verified crawler is exactly as automated as it says it is, and hiding is the one thing it is not doing.

Without that cap, the anomaly heuristics filed honest crawlers as attackers. Bingbot’s own renderer scored 91 and above, because its Chromium honestly reports navigator.webdriver and its fleet trips every datacenter, VPN and spoofed-UA check there is. Verified PerplexityBot and GPTBot page reads sat at 74 to 75.

Hostile acts bypass the cap. A honeypot trip or an attack signature is evidence of what the request did, not of what the client is, so a verified crawler that walks into a decoy is scored as the event it is.

Rotation across addresses is not evasion for a verified fleet

A crawler fleet is one identity operating from many addresses by design. WebDecoy’s rotation escalation, which treats a fingerprint seen from several addresses as an adversary evading IP blocks, is skipped for verified agents. Before that guard, range-verified Bingbot and ChatGPT-User fetching ordinary pages were escalated into HIGH and CRITICAL.

Spoofed and unverified claims still escalate, so a forged Googlebot rotating through residential ranges keeps scoring at the top.

Claiming an identity cannot buy a low score

The registry’s base scores run from 85 down to the twenties, and 44 catalogued agents carry a base of 40 or less and publish no IP ranges anybody can check: Slackbot, Twitterbot, Discordbot, WhatsApp, TelegramBot, LinkedInBot, UptimeRobot, Pingdom, the Wayback Machine, YandexBot, Baiduspider, Yahoo! Slurp and Sogou among them.

For all of those, granting the registry base unconditionally would mean that claiming the identity bought a cheap score. So an unverified claim gets the worse of the registry base and the generic heuristic. It can never be cheaper to claim an identity than to arrive without one. The rule only ever raises a score, so ByteSpider (unverified, base 75, heuristic 35) keeps its 75.

A published list that says nothing cannot condemn

Impersonation is proved by checking the address against the crawler’s own published ranges. OpenAI’s list for ChatGPT-User publishes 226 IPv4 prefixes and no IPv6 at all, so every IPv6 request carrying that user agent was being condemned as a forgery by a document that had never mentioned an address of that kind.

A range miss is now only reported when the published list could have contained the address. Otherwise the claim stays an unproven claim. Absence of an answer is not evidence of lying.

---

Known AI Crawlers

WebDecoy tracks and identifies the following AI crawlers:

Training Crawlers (High Score)

These crawlers collect content specifically for training AI/ML models:

Crawler	Company	Score	User-Agent Pattern	Purpose
GPTBot	OpenAI	85	GPTBot	Training GPT models
ClaudeBot	Anthropic	85	ClaudeBot	Training Claude models
Anthropic	Anthropic	85	anthropic-ai	Anthropic’s general crawler
CCBot	Common Crawl	80	CCBot	Open dataset used for AI training
Google-Extended	Google	80	Google-Extended	Gemini AI training
Cohere	Cohere	80	cohere-ai	Cohere model training
ByteSpider	ByteDance	75	Bytespider	TikTok and AI training
Meta-ExternalAgent	Meta	75	Meta-ExternalAgent	Meta AI training
Applebot-Extended	Apple	75	Applebot-Extended	Apple Intelligence training
YouBot	You.com	75	YouBot	You.com AI search training
Amazonbot	Amazon	70	Amazonbot	Alexa and Amazon AI
FacebookBot	Meta	70	FacebookBot	Meta AI training

AI Search Crawlers and AI Assistants

These are separate registry categories, ai_search_crawler and ai_assistant:

Crawler	Company	Category	Score	User-Agent Pattern
OAI-SearchBot	OpenAI	ai_search_crawler	80	OAI-SearchBot
PerplexityBot	Perplexity	ai_search_crawler	80	PerplexityBot
ChatGPT-User	OpenAI	ai_assistant	85	ChatGPT-User

Search Crawlers (Lower Score)

Traditional search engines that may also feed AI features:

Crawler	Company	Score	User-Agent Pattern	Note
Googlebot	Google	30	Googlebot	Primary search indexing
Bingbot	Microsoft	30	bingbot	Bing search + Copilot
DuckDuckBot	DuckDuckGo	30	DuckDuckBot	Privacy-focused search
Applebot	Apple	30	Applebot	Siri and Spotlight
YandexBot	Yandex	35	YandexBot	Russian search + AI
Baiduspider	Baidu	40	Baiduspider	Chinese search + AI

Search vs Training Crawlers

Search crawlers receive lower scores because they serve legitimate indexing purposes. However, many now also feed AI features (like Bing/Copilot), which blurs the line. Use the category field to distinguish between them.

These tables are a snapshot of the registry

The scores above come from the shared agent registry, which is the single source of truth for both scoring dimensions, the dashboard’s category filters, and the Node SDK’s local classifier. The registry catalogues 186 agents, considerably more than the tables here, and it gains entries as new crawlers appear. Treat these as the well-known cases rather than the whole list, and the category returned on a detection as authoritative.

---

AI Scraper Categories

A matched agent’s registry category is reported as ai_scraper_category:

Category	Registry base scores	Examples
training_crawler	65-85	GPTBot, ClaudeBot, CCBot, Google-Extended
ai_search_crawler	65-80	OAI-SearchBot, PerplexityBot
ai_assistant	60-85	ChatGPT-User
ai_agent	70-75	Autonomous AI agents
search_crawler	30-40	Googlebot, Bingbot, DuckDuckBot
seo_crawler	50-55	AhrefsBot, SemrushBot
security_scanner	65-75	Known vulnerability scanners
generic_scraper	50-65	python-requests, Scrapy
headless_browser	65-75	Headless browser user agents
commercial_scraper	65-70	Diffbot
fetcher	25-60	cURL, Wget, link-preview fetchers
archiver	25-30	Web archiving services
feed_reader	25	Feed readers
monitoring	5-25	Uptime monitors
none	0	No agent matched

Training Crawler

Crawlers that explicitly identify themselves as collecting data for AI/ML model training.

Characteristics:

• Honest self-identification in User-Agent
• Often respect robots.txt AI-specific directives
• High-volume, systematic crawling patterns
• Operate from known company IP ranges

robots.txt directive:

User-agent: GPTBot
Disallow: /

Search Crawler

Traditional search engine crawlers that index content for search results but may also feed AI features.

Consideration: While Googlebot itself scores low for AI scraping, content indexed by Google may be used in AI Overviews. Consider whether you want to limit Google’s AI-specific features via robots.txt.

None

Score: 0

No registry agent matched the User-Agent. This could be:

• Legitimate human traffic
• A bot spoofing a browser User-Agent (scored separately in Attack Risk)
• An AI crawler not yet in the registry

---

Threat Levels for AI Scraping

Score	Level	Interpretation	Recommended Action
0-19	MINIMAL	Not an AI scraper, or a low-impact agent such as a monitor	No action needed
20-39	LOW	Search crawler, archiver, feed reader	Log for analysis
40-59	MEDIUM	SEO crawler, generic scraper	Monitor and review
60-79	HIGH	AI search crawler, scanner, headless browser	Consider blocking
80-100	CRITICAL	Known AI training crawler or assistant	Consider blocking

---

Using AI Scraper Scores

At request time, in your app

The fastest way to act on this is the Node SDK, which carries the same agent registry locally and classifies the request before your handler runs: no network call, no waiting for the detection to be scored:

import { WebDecoy, bots } from '@webdecoy/node';

const wd = new WebDecoy({
  rules: [
    bots({ categories: ['training_crawler'] }),      // no AI training
    bots({ ai: true, allow: ['perplexitybot'] }),    // ...or all AI, minus one
  ],
});

The category names are identical to the ai_scraper_category values below, so a category you filter on in your dashboard is the one you write in a rule. See the Node SDK guide.

Bear in mind this matches on the declared User-Agent, which is exactly right for cooperative crawlers like GPTBot, and useless against anything spoofing a browser. For those, use tripwire().

Content Protection Strategy

The example below shows the decision shape given a scored detection, useful for offline analysis, alerting, or your own enforcement layer. For blocking in the request path, prefer the bots() rule above.

function handleRequest(detection) {
  const { unified_score } = detection;   // Attack risk (0-100)
  const {
    ai_scraper_score,     // AI scraper risk (0-100)
    ai_scraper_category,  // 'training_crawler', 'search_crawler', etc.
    ai_scraper_name       // 'GPTBot', 'ClaudeBot', etc.
  } = detection.score_components ?? {};

  // Block AI training crawlers from premium content
  if (ai_scraper_category === 'training_crawler' && ai_scraper_score >= 70) {
    return servePremiumContentBlocker();
  }

  // Allow search crawlers for SEO but log AI activity
  if (ai_scraper_category === 'search_crawler') {
    logAiCrawlerActivity(detection);
    return allowRequest();
  }

  // Challenge generic scraping tools
  if (ai_scraper_category === 'generic_scraper') {
    return challengeWithCaptcha();
  }

  // Handle attack threats separately
  if (unified_score >= 70) {
    return blockAttackRequest();
  }

  return allowRequest();
}

Tiered Content Access

Implement different content experiences based on AI scraper detection:

function getContentVersion(detection) {
  const { ai_scraper_score, ai_scraper_category } = detection.score_components ?? {};

  // AI training crawlers: Serve summary only
  if (ai_scraper_category === 'training_crawler') {
    return 'summary_only';
  }

  // Search crawlers: Full content for indexing
  if (ai_scraper_category === 'search_crawler') {
    return 'full_content';
  }

  // Suspicious scrapers: Serve paywall
  if (ai_scraper_score >= 50) {
    return 'paywall';
  }

  // Normal visitors: Full experience
  return 'full_content';
}

robots.txt is Not Enforcement

robots.txt is a request, not enforcement. Many AI crawlers ignore it entirely. WebDecoy gives you actual visibility into what’s being scraped, regardless of robots.txt. The Bot Directory shows whether each catalogued agent is known to respect robots.txt; WebDecoy does not compare individual requests against your robots.txt rules.

---

Dashboard Filtering

The AI Crawlers page (/verified-agents) is the report built for this: per-agent request counts, split into proven, unproven and forged claims. In the Detections table:

Filter	Effect
?ai_scraper_name=GPTBot in the URL, or the Agent cell’s filter control	Show only requests that claimed GPTBot
?agent_impersonation=true in the URL	Show only failed identity claims
Threat Category panel, Crawler or Scraper	Narrow by threat category

There is no filter on the AI scraper score or category itself.

Insights to Look For

1. Volume by Crawler: Which AI crawlers are hitting your site most?
2. Target Pages: Which content is being crawled most heavily? (see Crawled Pages)
3. Time Patterns: When do AI crawlers typically access your site?
4. Forged claims: Which requests claimed a crawler identity they could not prove?

---

Acting on AI Crawlers

Response actions trigger on a confirmed rotating actor, on any detection, or on a unified score threshold. None of them keys on the AI scraper score or category, and rate limiting is not a response action.

To act on AI crawlers by category, use the Node SDK’s bots() rule shown above, which runs in your request path. The framework adapters start in Monitor mode; blocking requires mode: 'enforce'.

---

API Access

Access AI Scraper data via the WebDecoy API:

Detection Response

The AI scraper fields live inside score_components on each detection. Illustrative example:

{
  "id": "5d0c2f7e-...",
  "ip_address": "20.15.240.128",
  "user_agent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0)",
  "requested_path": "/articles/exclusive-content.html",

  "unified_score": 14,
  "threat_level": "MINIMAL",
  "threat_category": "crawler",

  "score_components": {
    "honeypot_score": 0,
    "attack_signature_score": 0,
    "ai_scraper_score": 90,
    "ai_scraper_confidence": 0.95,
    "ai_scraper_category": "training_crawler",
    "ai_scraper_name": "GPTBot"
  }
}

Filtering by AI Scraper

The management API takes the Auth0 session token the dashboard uses, not an sk_live_ API key (see API Reference):

# Every detection that claimed to be GPTBot
curl -H "Authorization: Bearer $TOKEN" \
  "https://api.webdecoy.com/api/organizations/$ORG_ID/detections?ai_scraper_name=GPTBot"

# AI scraper activity per crawler
curl -H "Authorization: Bearer $TOKEN" \
  "https://api.webdecoy.com/api/organizations/$ORG_ID/detections/stats/ai-scrapers/by-name"

---

Future Considerations

Expanding Crawler Detection

The AI landscape is evolving rapidly. WebDecoy regularly updates its crawler database as new AI training bots emerge. Factors we monitor:

• New AI Companies: Startups launching training crawlers
• Existing Companies: Tech giants adding AI features to existing bots
• IP Range Updates: Changes to AI company infrastructure
• Behavioral Patterns: New scraping techniques and evasion methods

Legal and Licensing Implications

AI scraping raises significant copyright and licensing questions. WebDecoy provides the data layer for understanding what’s being scraped. This can support:

• Licensing negotiations with AI companies
• Copyright enforcement evidence
• Terms of service violation documentation

---

Summary

Aspect	Attack Scoring	AI Scraper Scoring
Question	Is this an attack or bot?	Is this an AI training crawler?
User-Agent Weight	1% (easily spoofed)	Registry base score, up to 85 (reliable ID)
Interpretation	Honest ID = low threat	Honest ID = high threat
Goal	Detect attacks and evasion	Detect content harvesting
Primary Users	Security teams	Publishers, content creators

Getting Started

To start monitoring AI scraper activity:

1. Deploy WebDecoy on your domain
2. View the Detections dashboard
3. Open AI Crawlers to see which crawlers are accessing your content, and whether their claims are proven
4. Act on them in your request path with the Node SDK’s bots() rule

---

Related Documentation

• Threat Scoring - Attack and bot detection scoring
• Response Actions - Automated responses to threats
• Detection Script - JavaScript-based bot detection