AI Scraper Scoring

WebDecoy includes a dedicated scoring dimension for detecting AI training crawlers. This runs completely independently from attack detection because the signal interpretation is fundamentally different.

The Problem: AI Scrapers Don’t Look Like Attackers

When GPTBot crawls your website, it honestly identifies itself in the User-Agent header. From an attack detection perspective, this is low threat: the bot isn’t trying to hide or evade detection.

But from a content protection perspective, this is the highest possible threat: a known AI training crawler is actively collecting your content.

Same signal, opposite interpretations:

┌─────────────────────────────────────────────────────────────────┐
│                        GPTBot Request                            │
│   User-Agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like       │
│   Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)    │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│   Attack Detection View          │   Content Protection View     │
│   ──────────────────────         │   ────────────────────────    │
│   "This bot isn't hiding"        │   "Known AI training crawler" │
│   User-Agent easily spoofed      │   Confirmed identity          │
│   Score: 14 (MINIMAL)            │   Score: 85 (CRITICAL)        │
│   Verdict: Low threat            │   Verdict: High threat        │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Why Two Separate Scores?

WebDecoy shows both scores side-by-side because they answer different questions:

Score	Question It Answers
Attack Risk	“Is this visitor trying to attack or evade detection?”
AI Scraper Risk	“Is this visitor an AI crawler collecting content for training?”

For a detected GPTBot request:

┌─────────────────────────┬─────────────────────────┐
│      Attack Risk        │    AI Scraper Risk      │
│                         │                         │
│          14             │           85            │
│       MINIMAL           │        CRITICAL         │
│                         │                         │
│  "Not trying to hide"   │   "GPTBot detected"     │
└─────────────────────────┴─────────────────────────┘

Key insight: Honest self-identification is LOW threat for attack detection but HIGH confidence for AI scraper detection.

When the identity is not merely honest but verified, the split widens by design: the attack score is capped at 60 absent a hostile act, while the AI scraper score is unchanged. See What verification does to each score.

Publisher Value

For publishers and content creators, AI Scraper scores provide the visibility that attack scores alone cannot deliver. This is the core value proposition for content protection.

---

How AI Scraper Scoring Works

Inverted Signal Weighting

AI Scraper scoring uses inverted signal weighting compared to attack detection:

Signal	Attack Scoring	AI Scraper Scoring	Reasoning
Honest bot UA (GPTBot, ClaudeBot)	1% weight (trivially spoofed)	85+ points (confirmed identity)	For attack detection, UA can be spoofed. For AI detection, honest self-ID is reliable
Datacenter IP	3% weight (VPN false positives)	+10 points boost	AI crawlers typically operate from cloud infrastructure
Missing Referer	1% weight (easily spoofed)	+5 points boost	Direct access patterns are typical for crawlers
robots.txt access	Not scored	Tracked for compliance analysis	Shows whether crawler is checking robots.txt before crawling

Both dimensions read the same registry

The crawler tables below are not a second list maintained beside the attack scorer’s. Both dimensions ask the shared agent registry what a client is, and take its answer.

That was not always true on the attack side. The attack scorer used to derive its User-Agent component from a substring rule: any User-Agent containing bot scored 100. That rule is blind in both directions. ChatGPT-User matched on +https://openai.com/bot, the courtesy URL a crawler publishes precisely so it can be identified, and scored 100. ByteSpider contains no bot at all, fell through to a generic suspicious-UA rule, and scored 35 against a registry base of 75, despite being the only agent in that comparison that ignores robots.txt.

Both dimensions now sit in the order the registry assigns. If you filter your dashboard on a category name, write a bots() rule against it in the Node SDK, and read an attack score for the same request, all three are describing the same table.

Detection Process

┌─────────────────────────────────────────────────────────────────┐
│                 AI SCRAPER DETECTION PIPELINE                    │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│   Incoming Request                                               │
│         │                                                        │
│         ▼                                                        │
│   ┌───────────────────────────────────────────┐                 │
│   │     CHECK USER-AGENT PATTERNS             │                 │
│   │  GPTBot? ClaudeBot? CCBot? ByteSpider?    │                 │
│   └─────────────────┬─────────────────────────┘                 │
│                     │                                            │
│        ┌────────────┴────────────┐                              │
│        │                         │                               │
│        ▼                         ▼                               │
│   ┌─────────┐              ┌───────────┐                        │
│   │ MATCHED │              │ NO MATCH  │                        │
│   └────┬────┘              └─────┬─────┘                        │
│        │                         │                               │
│        ▼                         ▼                               │
│   ┌───────────────┐        ┌───────────────┐                    │
│   │ Assign Score  │        │ Check Other   │                    │
│   │ + Confidence  │        │ Signals       │                    │
│   │ + Category    │        │ (behavioral)  │                    │
│   └───────┬───────┘        └───────┬───────┘                    │
│           │                        │                             │
│           └──────────┬─────────────┘                             │
│                      ▼                                           │
│            ┌─────────────────┐                                   │
│            │  AI Scraper     │                                   │
│            │  Score + Level  │                                   │
│            │  + Category     │                                   │
│            └─────────────────┘                                   │
└─────────────────────────────────────────────────────────────────┘

Scoring Components

The AI Scraper Score is calculated from multiple signals:

Signal	Points	Confidence Boost
Known Training Crawler UA	70-85	95%
Known Search Crawler UA	30-40	90%
AI Company IP Block	+10-20	+10%
Datacenter/Cloud IP	+5-10	none
High Request Rate Pattern	+5-15	+5%
Systematic URL Pattern	+5-10	none
Missing Referer	+5	none

---

What verification does to each score

A crawler’s declared identity can be checked: against the IP ranges the operator publishes, against a forward-confirmed reverse DNS record, or against a Web Bot Auth signature. The outcome of that check pulls the two dimensions in opposite directions, which is the clearest illustration there is of why they are separate numbers.

Verification outcome	AI Scraper Risk	Attack Risk
Verified	Unchanged. It really is GPTBot, and it really is reading your content	Capped at 60, the top of MEDIUM, absent a hostile act
Unverified	Unchanged. The claim is still worth logging	Scored on its own merits: the worse of the registry base and the generic heuristic
Spoofed	The name is reported with the claim marked as failed	Hits the impersonation floor

A verified agent is capped at the top of MEDIUM

The attack dimension measures how automated and how evasive a client looks. A proven identity answers both questions at once: a verified crawler is exactly as automated as it says it is, and hiding is the one thing it is not doing.

Without that cap, the anomaly heuristics filed honest crawlers as attackers. Bingbot’s own renderer scored 91 and above, because its Chromium honestly reports navigator.webdriver and its fleet trips every datacenter, VPN and spoofed-UA check there is. Verified PerplexityBot and GPTBot page reads sat at 74 to 75.

Hostile acts bypass the cap. A honeypot trip or an attack signature is evidence of what the request did, not of what the client is, so a verified crawler that walks into a decoy is scored as the event it is.

Rotation across addresses is not evasion for a verified fleet

A crawler fleet is one identity operating from many addresses by design. WebDecoy’s rotation escalation, which treats a fingerprint seen from several addresses as an adversary evading IP blocks, is skipped for verified agents. Before that guard, roughly 2,400 detections over 14 days from range-verified and rDNS-verified agents were pushed into HIGH and CRITICAL, including 1,565 verified ChatGPT-User requests for ordinary blog pages.

Spoofed and unverified claims still escalate, so a forged Googlebot rotating through residential ranges keeps scoring at the top.

Claiming an identity cannot buy a low score

The registry’s base scores run from 85 down to the twenties, and 44 catalogued agents carry a base of 40 or less and publish no IP ranges anybody can check: Slackbot, Twitterbot, Discordbot, WhatsApp, TelegramBot, LinkedInBot, UptimeRobot, Pingdom, the Wayback Machine, YandexBot, Baiduspider, Yahoo! Slurp and Sogou among them.

For all of those, granting the registry base unconditionally would mean that claiming the identity bought a cheap score. So an unverified claim gets the worse of the registry base and the generic heuristic. It can never be cheaper to claim an identity than to arrive without one. The rule only ever raises a score, so ByteSpider (unverified, base 75, heuristic 35) keeps its 75.

A published list that says nothing cannot condemn

Impersonation is proved by checking the address against the crawler’s own published ranges. OpenAI’s list for ChatGPT-User publishes 226 IPv4 prefixes and no IPv6 at all, so every IPv6 request carrying that user agent was being condemned as a forgery by a document that had never mentioned an address of that kind.

A range miss is now only reported when the published list could have contained the address. Otherwise the claim stays an unproven claim. Absence of an answer is not evidence of lying.

---

Known AI Crawlers

WebDecoy tracks and identifies the following AI crawlers:

Training Crawlers (High Score)

These crawlers collect content specifically for training AI/ML models:

Crawler	Company	Score	User-Agent Pattern	Purpose
GPTBot	OpenAI	85	GPTBot/1.0	Training ChatGPT and GPT models
ChatGPT-User	OpenAI	85	ChatGPT-User	ChatGPT plugins and browsing
OAI-SearchBot	OpenAI	80	OAI-SearchBot	SearchGPT content retrieval
ClaudeBot	Anthropic	85	ClaudeBot	Training Claude models
Anthropic	Anthropic	85	anthropic-ai	Anthropic’s general crawler
CCBot	Common Crawl	80	CCBot/2.0	Open dataset for AI training
Google-Extended	Google	80	Google-Extended	Gemini/Bard AI training
PerplexityBot	Perplexity	80	PerplexityBot	Perplexity AI search
Cohere	Cohere	80	cohere-ai	Cohere model training
ByteSpider	ByteDance	75	Bytespider	TikTok/Douyin AI features
Meta-ExternalAgent	Meta	75	Meta-ExternalAgent	Meta AI training
Applebot-Extended	Apple	75	Applebot-Extended	Apple Intelligence training
YouBot	You.com	75	YouBot	You.com AI search
Amazonbot	Amazon	70	Amazonbot	Alexa and Amazon AI
FacebookBot	Meta	70	facebookexternalhit	Facebook AI features
Diffbot	Diffbot	70	Diffbot	Knowledge graph extraction

Search Crawlers (Lower Score)

Traditional search engines that may also feed AI features:

Crawler	Company	Score	User-Agent Pattern	Note
Googlebot	Google	30	Googlebot	Primary search indexing
Bingbot	Microsoft	30	bingbot	Bing search + Copilot
DuckDuckBot	DuckDuckGo	30	DuckDuckBot	Privacy-focused search
Applebot	Apple	30	Applebot	Siri and Spotlight
YandexBot	Yandex	35	YandexBot	Russian search + AI
Baiduspider	Baidu	40	Baiduspider	Chinese search + AI

Search vs Training Crawlers

Search crawlers receive lower scores because they serve legitimate indexing purposes. However, many now also feed AI features (like Bing/Copilot), which blurs the line. Use the category field to distinguish between them.

These tables are a snapshot of the registry

The scores above come from the shared agent registry, which is the single source of truth for both scoring dimensions, the dashboard’s category filters, and the Node SDK’s local classifier. The registry catalogues considerably more agents than the two tables here, and it gains entries as new crawlers appear. Treat these as the well-known cases rather than the whole list, and the category returned on a detection as authoritative.

---

AI Scraper Categories

Detected AI activity is classified into categories:

Training Crawler

Score Range: 70-85

Crawlers that explicitly identify themselves as collecting data for AI/ML model training.

Characteristics:

• Honest self-identification in User-Agent
• Often respect robots.txt AI-specific directives
• High-volume, systematic crawling patterns
• Operate from known company IP ranges

Examples: GPTBot, ClaudeBot, CCBot, Google-Extended

robots.txt directive:

User-agent: GPTBot
Disallow: /

---

Search Crawler

Score Range: 30-40

Traditional search engine crawlers that index content for search results but may also feed AI features.

Characteristics:

• Well-established crawling behavior
• Generally respect robots.txt
• Verifiable by IP (e.g., Googlebot verification)
• Lower threat to content licensing than pure AI trainers

Examples: Googlebot, Bingbot, DuckDuckBot

Consideration: While Googlebot itself scores low for AI scraping, content indexed by Google may be used in AI Overviews. Consider whether you want to limit Google’s AI-specific features via robots.txt.

---

Content Scraper

Score Range: Variable (based on behavioral signals)

Scrapers detected by behavior rather than self-identification. These don’t declare themselves as AI crawlers but exhibit scraping patterns.

Characteristics:

• May spoof User-Agent as regular browser
• Detected by fingerprint anomalies
• High request rates
• Systematic URL patterns
• Missing typical browser behaviors

Detection signals:

• Headless browser fingerprint
• No mouse/keyboard events
• Impossibly fast navigation
• Sequential URL access patterns

---

None

Score Range: 0

Not detected as an AI scraper. This could be:

• Legitimate human traffic
• Attack bot (scored separately in Attack Risk)
• Unknown/new AI crawler not yet in database

---

Threat Levels for AI Scraping

Score	Level	Interpretation	Recommended Action
0-20	MINIMAL	Not an AI scraper	No action needed
21-40	LOW	Search crawler or weak signals	Log for analysis
41-60	MEDIUM	Possible AI scraper activity	Monitor and review
61-80	HIGH	Strong AI scraper indicators	Consider blocking
81-100	CRITICAL	Confirmed AI training crawler	Block or serve alternative content

---

Using AI Scraper Scores

At request time, in your app

The fastest way to act on this is the Node SDK, which carries the same agent registry locally and classifies the request before your handler runs: no network call, no waiting for the detection to be scored:

import { WebDecoy, bots } from '@webdecoy/node';

const wd = new WebDecoy({
  rules: [
    bots({ categories: ['training_crawler'] }),      // no AI training
    bots({ ai: true, allow: ['perplexitybot'] }),    // ...or all AI, minus one
  ],
});

The category names are identical to the ai_scraper_category values below, so a category you filter on in your dashboard is the one you write in a rule. See the Node SDK guide.

Bear in mind this matches on the declared User-Agent, which is exactly right for cooperative crawlers like GPTBot, and useless against anything spoofing a browser. For those, use tripwire().

Content Protection Strategy

The example below shows the decision shape given a scored detection, useful for offline analysis, alerting, or your own enforcement layer. For blocking in the request path, prefer the bots() rule above.

function handleRequest(detection) {
  const {
    unified_score,        // Attack risk (0-100)
    ai_scraper_score,     // AI scraper risk (0-100)
    ai_scraper_category,  // 'training_crawler', 'search_crawler', etc.
    ai_scraper_name       // 'GPTBot', 'ClaudeBot', etc.
  } = detection;

  // Block AI training crawlers from premium content
  if (ai_scraper_category === 'training_crawler' && ai_scraper_score >= 70) {
    return servePremiumContentBlocker();
  }

  // Allow search crawlers for SEO but log AI activity
  if (ai_scraper_category === 'search_crawler') {
    logAiCrawlerActivity(detection);
    return allowRequest();
  }

  // Challenge unknown scrapers
  if (ai_scraper_score >= 50 && ai_scraper_category === 'content_scraper') {
    return challengeWithCaptcha();
  }

  // Handle attack threats separately
  if (unified_score >= 70) {
    return blockAttackRequest();
  }

  return allowRequest();
}

Tiered Content Access

Implement different content experiences based on AI scraper detection:

function getContentVersion(detection) {
  const { ai_scraper_score, ai_scraper_category } = detection;

  // AI training crawlers: Serve summary only
  if (ai_scraper_category === 'training_crawler') {
    return 'summary_only';
  }

  // Search crawlers: Full content for indexing
  if (ai_scraper_category === 'search_crawler') {
    return 'full_content';
  }

  // Suspicious scrapers: Serve paywall
  if (ai_scraper_score >= 50) {
    return 'paywall';
  }

  // Normal visitors: Full experience
  return 'full_content';
}

robots.txt Compliance Monitoring

WebDecoy can track whether AI crawlers are respecting your robots.txt directives:

Your robots.txt:
─────────────────
User-agent: GPTBot
Disallow: /premium/
Disallow: /articles/

WebDecoy Detection:
──────────────────
GPTBot accessed: /articles/2024/my-exclusive-story.html
AI Scraper Score: 85
robots.txt Compliance: VIOLATED

→ This crawler is ignoring your robots.txt!

robots.txt is Not Enforcement

robots.txt is a request, not enforcement. Many AI crawlers ignore it entirely. WebDecoy gives you actual visibility into what’s being scraped, regardless of robots.txt compliance.

---

Dashboard Filtering

In the Detections table, filter by AI scraper activity:

Filter Options

Filter	Effect
AI Scraper Score >= 70	Show high-confidence AI crawler detections
Category = training_crawler	Show only AI training crawlers
AI Scraper Name = GPTBot	Show only GPTBot activity
AI Scraper Score > 0	Show all AI scraper activity

Insights to Look For

1. Volume by Crawler: Which AI crawlers are hitting your site most?
2. Target Pages: Which content is being scraped most heavily?
3. Time Patterns: When do AI crawlers typically access your site?
4. robots.txt Violations: Are crawlers ignoring your preferences?
5. New Crawlers: Unknown scraping patterns that might be new AI crawlers

---

Integration with Response Actions

Use AI Scraper scores to trigger automated responses:

Block AI Training Crawlers

Response Action: Block AI Scrapers
Trigger: ai_scraper_score >= 70 AND ai_scraper_category = 'training_crawler'
Action: Block
Duration: Permanent
Notify: Weekly digest

Rate Limit Search Crawlers

Response Action: Rate Limit Search Crawlers
Trigger: ai_scraper_category = 'search_crawler'
Action: Rate Limit
Rate: 100 requests/hour
Notify: On threshold breach

Alert on High-Volume Scraping

Response Action: Scraping Alert
Trigger: ai_scraper_score >= 50 AND request_count > 1000/hour
Action: Webhook Alert
Endpoint: https://your-api.com/alerts
Notify: Immediately

---

API Access

Access AI Scraper data via the WebDecoy API:

Detection Response

{
  "id": "det_abc123",
  "ip_address": "20.15.240.128",
  "user_agent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0)",
  "path": "/articles/exclusive-content.html",

  "unified_score": 14,
  "threat_level": "MINIMAL",
  "category": "crawler",

  "ai_scraper_score": 85,
  "ai_scraper_level": "CRITICAL",
  "ai_scraper_category": "training_crawler",
  "ai_scraper_name": "GPTBot",
  "ai_scraper_confidence": 0.95,

  "score_components": {
    "user_agent_score": 100,
    "honeypot_score": 0,
    "attack_signature_score": 0,
    "ai_scraper_score": 85,
    "ai_scraper_confidence": 0.95,
    "ai_scraper_category": "training_crawler",
    "ai_scraper_name": "GPTBot"
  }
}

Filtering by AI Scraper

# Get all AI training crawler detections
curl -H "Authorization: Bearer $TOKEN" \
  "https://api.webdecoy.com/v1/detections?ai_scraper_category=training_crawler"

# Get high-score AI scraper activity
curl -H "Authorization: Bearer $TOKEN" \
  "https://api.webdecoy.com/v1/detections?ai_scraper_score_gte=70"

---

Future Considerations

Expanding Crawler Detection

The AI landscape is evolving rapidly. WebDecoy regularly updates its crawler database as new AI training bots emerge. Factors we monitor:

• New AI Companies: Startups launching training crawlers
• Existing Companies: Tech giants adding AI features to existing bots
• IP Range Updates: Changes to AI company infrastructure
• Behavioral Patterns: New scraping techniques and evasion methods

AI-Specific Content Directives

Some sites are experimenting with AI-specific content tags:

<!-- Potential future standard -->
<meta name="ai-training" content="disallow">
<meta name="ai-summary" content="allow">

WebDecoy will track adoption and compliance with emerging standards.

Legal and Licensing Implications

AI scraping raises significant copyright and licensing questions. WebDecoy provides the data layer for understanding what’s being scraped. This can support:

• Licensing negotiations with AI companies
• Copyright enforcement evidence
• Terms of service violation documentation
• Opt-out compliance verification

---

Summary

Aspect	Attack Scoring	AI Scraper Scoring
Question	Is this an attack or bot?	Is this an AI training crawler?
User-Agent Weight	1% (easily spoofed)	70-85+ points (reliable ID)
Interpretation	Honest ID = low threat	Honest ID = high threat
Goal	Detect attacks and evasion	Detect content harvesting
Primary Users	Security teams	Publishers, content creators

Getting Started

To start monitoring AI scraper activity:

1. Deploy WebDecoy on your domain
2. View the Detections dashboard
3. Filter by “AI Scraper Score >= 50”
4. Analyze which crawlers are accessing your content
5. Configure Response Actions based on your content protection policy

---

Related Documentation

• Threat Scoring - Attack and bot detection scoring
• Response Actions - Automated responses to threats
• Detection Script - JavaScript-based bot detection