Good Bots & Verified Crawlers

Use this good-bot list as a reference for legitimate search engines, AI crawlers, monitoring services, and social preview bots recognized by WebDecoy. Recognition labels a crawler; verification proves that the request actually came from its operator.

---

Overview

WebDecoy recognizes known legitimate bots for classification, and (where it matters most) verifies them before letting them through enforcement.

Two distinct things use this list:

1. Detection classification (informational): recognized bots are labeled by category in your detections so you can tell Googlebot from a scraper at a glance.
2. The enforcement allowlist (gating): on token-enforced routes, exactly three verified categories can pass the clearance gate without a token: search engines, AI crawlers, and monitoring services. Each is verified against operator-published IP ranges plus forward-confirmed reverse DNS; a cloud VM claiming to be Googlebot fails both checks. There is deliberately no manual user-agent or IP allowlist: proof, not provenance. Your own machine clients use machine service tokens instead.

The tables below are the informational reference of bots by category.

A JavaScript-only install cannot see most of these

Nearly every bot on this page (Googlebot’s crawl pass, GPTBot, ClaudeBot, CCBot, uptime monitors, curl) does not execute JavaScript. The detection script runs in the visitor’s browser, so it structurally cannot observe them, and no amount of tuning changes that.

If your only detection source is the page tag, expect these categories to stay empty. Seeing them requires a source in front of your origin: the Cloudflare Edge Sensor, a server-side SDK such as Express or Next.js, or the WordPress plugin.

Bots that do run JavaScript (headless browsers, some social preview fetchers) are visible to the page tag as normal.

---

Search Engine Bots

Primary search engine crawlers that index your content.

Bot	User Agent Pattern	Purpose
Googlebot	Googlebot	Google Search indexing
Googlebot Images	Googlebot-Image	Google Images indexing
Googlebot Video	Googlebot-Video	Google Video indexing
Googlebot News	Googlebot-News	Google News indexing
Google AdsBot	AdsBot-Google	Google Ads landing page check
Bingbot	bingbot	Bing Search indexing
Yahoo! Slurp	Slurp	Yahoo Search indexing
DuckDuckBot	DuckDuckBot	DuckDuckGo indexing
Baiduspider	Baiduspider	Baidu (China) indexing
YandexBot	YandexBot	Yandex (Russia) indexing
Sogou	Sogou	Sogou (China) indexing
Exabot	Exabot	Exalead indexing
Qwantify	Qwantify	Qwant indexing

Verification

Google and Bing bots can be verified by reverse DNS:

• Googlebot: *.googlebot.com or *.google.com
• Bingbot: *.search.msn.com

---

Social Media Bots

Crawlers that generate link previews and share content.

Bot	User Agent Pattern	Purpose
Facebook	facebookexternalhit	Link preview generation
Facebook Catalog	Facebot	Product catalog scraping
Twitter	Twitterbot	Tweet card generation
LinkedIn	LinkedInBot	Post preview generation
Pinterest	Pinterest	Pin image fetching
WhatsApp	WhatsApp	Link preview generation
Slack	Slackbot	Link unfurling
Telegram	TelegramBot	Link preview
Discord	Discordbot	Embed generation
Skype	SkypeUriPreview	Link preview

---

AI/LLM Crawlers

Crawlers for AI training and LLM services.

Bot	User Agent Pattern	Purpose	Default
GPTBot	GPTBot	OpenAI training	Blocked
ChatGPT-User	ChatGPT-User	ChatGPT browsing	Blocked
ClaudeBot	ClaudeBot, Claude-Web	Anthropic training	Blocked
Google-Extended	Google-Extended	Google AI training	Blocked
PerplexityBot	PerplexityBot	Perplexity AI	Blocked
CCBot	CCBot	Common Crawl	Blocked
Cohere	cohere-ai	Cohere training	Blocked
Applebot Extended	Applebot-Extended	Apple AI features	Blocked

Note: AI crawlers are blocked by default. Enable them in Settings if you want your content included in AI training.

---

Monitoring & Uptime Bots

Services that check your site availability.

Bot	User Agent Pattern	Purpose
Pingdom	Pingdom	Uptime monitoring
UptimeRobot	UptimeRobot	Uptime monitoring
StatusCake	StatusCake	Uptime monitoring
Site24x7	Site24x7	Performance monitoring
Datadog	Datadog	APM & monitoring
New Relic	NewRelicPinger	Performance monitoring
GTmetrix	GTmetrix	Performance testing
WebPageTest	WebPageTest	Performance testing
Catchpoint	Catchpoint	Synthetic monitoring

---

SEO & Marketing Bots

Tools used for SEO analysis and marketing.

Bot	User Agent Pattern	Purpose
Semrush	SemrushBot	SEO analysis
Ahrefs	AhrefsBot	Backlink analysis
Moz	MozBot, rogerbot	SEO analysis
Majestic	MJ12bot	Backlink analysis
Screaming Frog	Screaming Frog	Site crawling
Sistrix	SISTRIX	SEO analysis
Serpstat	serpstatbot	SEO analysis
SpyFu	SpyFu	Competitor analysis

---

Feed & Content Aggregators

Services that fetch RSS/Atom feeds and aggregate content.

Bot	User Agent Pattern	Purpose
Feedly	Feedly	RSS aggregation
Feedbin	Feedbin	RSS reader
Inoreader	Inoreader	RSS reader
NewsBlur	NewsBlur	RSS reader
Apple News	AppleNewsBot	Apple News aggregation
Flipboard	Flipboard	Content aggregation

---

Developer & Testing Tools

Tools used for website testing and development.

Bot	User Agent Pattern	Purpose
W3C Validator	W3C_Validator	HTML validation
W3C Link Checker	W3C-checklink	Link validation
Google PageSpeed	Google Page Speed	Performance testing
Lighthouse	Chrome-Lighthouse	Web auditing
Archive.org	ia_archiver	Web archiving

---

Infrastructure Bots

Cloud provider and CDN health checks.

Bot	User Agent Pattern	Purpose
AWS ELB	ELB-HealthChecker	Load balancer health
Cloudflare	CloudFlare-AlwaysOnline	Always Online feature
Fastly	Fastly	CDN health check
Akamai	AkamaiGHost	CDN monitoring

---

Configuration

The enforcement allowlist

On the Enforcement page (step 4, “Allow tokenless clients”), three toggles control which verified crawler categories pass the clearance gate on enforced routes:

Category	Examples	Verified how
Search engines	Googlebot, Bingbot	Published IP ranges + forward-confirmed rDNS
AI crawlers	GPTBot, ClaudeBot, PerplexityBot	Published IP ranges + rDNS
Monitoring services	Verified uptime/synthetics providers	Published ranges

Turning a category off challenges those crawlers on enforced routes. There is no custom user-agent allowlist: your own bots, partners, and jobs authenticate with per-client machine service tokens minted on the same page (revocable individually, no shared blast radius).

(The WordPress plugin has its own local Good Bots settings. See the plugin guide.)

Verification Methods

For critical applications, verify bot identity:

1. Reverse DNS (recommended for Google/Bing)

   host <ip_address>
   # Should resolve to *.googlebot.com or *.search.msn.com

2. IP Range Verification

   • Google publishes Googlebot IP ranges
   • Bing publishes bingbot IP ranges

---

Bot Identification in Detections

When a good bot is detected:

{
  "is_good_bot": true,
  "bot_name": "Googlebot",
  "bot_category": "search_engine",
  "bot_verified": true,
  "threat_score": 5
}

Bot Categories

These are detection classification categories (informational labels on detections). Only search_engine, ai_crawler, and monitoring correspond to verified enforcement-allowlist categories.

Category	Examples
search_engine	Googlebot, Bingbot
social_media	Facebook, Twitter
ai_crawler	GPTBot, ClaudeBot
monitoring	Pingdom, UptimeRobot
seo_tool	Semrush, Ahrefs
feed_reader	Feedly, Feedbin
unknown	Unrecognized bot