Good Bots & Verified Crawlers
Use this good-bot list as a reference for legitimate search engines, AI crawlers, monitoring services, and social preview bots recognized by WebDecoy. Recognition labels a crawler; verification proves that the request actually came from its operator.
Overview
Section titled “Overview”WebDecoy recognizes known legitimate bots for classification, and (where it matters most) verifies them before letting them through enforcement.
Two distinct things use this list:
- Detection classification (informational): recognized bots are labeled by category in your detections so you can tell Googlebot from a scraper at a glance.
- Access exceptions (gating): on token-enforced routes, verified bots whose behavior you tick (for example search, AI training crawlers, link previews, uptime and analytics) can pass the clearance gate without a token. Verification uses operator-published IP ranges, forward-confirmed reverse DNS, or a cryptographic signature; a cloud VM claiming to be Googlebot fails. There is deliberately no manual user-agent or IP allowlist: proof, not provenance. Your own machine clients use machine service tokens instead.
The tables below are the informational reference of bots by category.
Search Engine Bots
Section titled “Search Engine Bots”Primary search engine crawlers that index your content.
| Bot | User Agent Pattern | Purpose |
|---|---|---|
| Googlebot | Googlebot |
Google Search indexing |
| Googlebot Images | Googlebot-Image |
Google Images indexing |
| Googlebot Video | Googlebot-Video |
Google Video indexing |
| Googlebot News | Googlebot-News |
Google News indexing |
| Google AdsBot | AdsBot-Google |
Google Ads landing page check |
| Bingbot | bingbot |
Bing Search indexing |
| Yahoo! Slurp | Slurp |
Yahoo Search indexing |
| DuckDuckBot | DuckDuckBot |
DuckDuckGo indexing |
| Baiduspider | Baiduspider |
Baidu (China) indexing |
| YandexBot | YandexBot |
Yandex (Russia) indexing |
| Sogou | Sogou |
Sogou (China) indexing |
| Exabot | Exabot |
Exalead indexing |
| Qwantify | Qwantify |
Qwant indexing |
Verification
Section titled “Verification”Google and Bing bots can be verified by reverse DNS:
- Googlebot:
*.googlebot.comor*.google.com - Bingbot:
*.search.msn.com
Social Media Bots
Section titled “Social Media Bots”Crawlers that generate link previews and share content.
| Bot | User Agent Pattern | Purpose |
|---|---|---|
facebookexternalhit |
Link preview generation | |
| Facebook Catalog | Facebot |
Product catalog scraping |
Twitterbot |
Tweet card generation | |
LinkedInBot |
Post preview generation | |
Pinterest |
Pin image fetching | |
WhatsApp |
Link preview generation | |
| Slack | Slackbot |
Link unfurling |
| Telegram | TelegramBot |
Link preview |
| Discord | Discordbot |
Embed generation |
| Skype | SkypeUriPreview |
Link preview |
AI/LLM Crawlers
Section titled “AI/LLM Crawlers”Crawlers for AI training and LLM services.
| Bot | User Agent Pattern | Purpose |
|---|---|---|
| GPTBot | GPTBot |
OpenAI training |
| ChatGPT-User | ChatGPT-User |
ChatGPT browsing |
| ClaudeBot | ClaudeBot, Claude-Web |
Anthropic training |
| Google-Extended | Google-Extended |
Google AI training |
| PerplexityBot | PerplexityBot |
Perplexity AI |
| CCBot | CCBot |
Common Crawl |
| Cohere | cohere-ai |
Cohere training |
| Applebot Extended | Applebot-Extended |
Apple AI features |
Note: WebDecoy does not block AI crawlers by default. The enforcement exceptions let verified crawlers through unless you untick them, and the WordPress plugin’s Block AI crawlers option is off by default.
Monitoring & Uptime Bots
Section titled “Monitoring & Uptime Bots”Services that check your site availability.
| Bot | User Agent Pattern | Purpose |
|---|---|---|
| Pingdom | Pingdom |
Uptime monitoring |
| UptimeRobot | UptimeRobot |
Uptime monitoring |
| StatusCake | StatusCake |
Uptime monitoring |
| Site24x7 | Site24x7 |
Performance monitoring |
| Datadog | Datadog |
APM & monitoring |
| New Relic | NewRelicPinger |
Performance monitoring |
| GTmetrix | GTmetrix |
Performance testing |
| WebPageTest | WebPageTest |
Performance testing |
| Catchpoint | Catchpoint |
Synthetic monitoring |
SEO & Marketing Bots
Section titled “SEO & Marketing Bots”Tools used for SEO analysis and marketing.
| Bot | User Agent Pattern | Purpose |
|---|---|---|
| Semrush | SemrushBot |
SEO analysis |
| Ahrefs | AhrefsBot |
Backlink analysis |
| Moz | MozBot, rogerbot |
SEO analysis |
| Majestic | MJ12bot |
Backlink analysis |
| Screaming Frog | Screaming Frog |
Site crawling |
| Sistrix | SISTRIX |
SEO analysis |
| Serpstat | serpstatbot |
SEO analysis |
| SpyFu | SpyFu |
Competitor analysis |
Feed & Content Aggregators
Section titled “Feed & Content Aggregators”Services that fetch RSS/Atom feeds and aggregate content.
| Bot | User Agent Pattern | Purpose |
|---|---|---|
| Feedly | Feedly |
RSS aggregation |
| Feedbin | Feedbin |
RSS reader |
| Inoreader | Inoreader |
RSS reader |
| NewsBlur | NewsBlur |
RSS reader |
| Apple News | AppleNewsBot |
Apple News aggregation |
Flipboard |
Content aggregation |
Developer & Testing Tools
Section titled “Developer & Testing Tools”Tools used for website testing and development.
| Bot | User Agent Pattern | Purpose |
|---|---|---|
| W3C Validator | W3C_Validator |
HTML validation |
| W3C Link Checker | W3C-checklink |
Link validation |
| Google PageSpeed | Google Page Speed |
Performance testing |
| Lighthouse | Chrome-Lighthouse |
Web auditing |
| Archive.org | ia_archiver |
Web archiving |
Infrastructure Bots
Section titled “Infrastructure Bots”Cloud provider and CDN health checks.
| Bot | User Agent Pattern | Purpose |
|---|---|---|
| AWS ELB | ELB-HealthChecker |
Load balancer health |
| Cloudflare | CloudFlare-AlwaysOnline |
Always Online feature |
| Fastly | Fastly |
CDN health check |
| Akamai | AkamaiGHost |
CDN monitoring |
Configuration
Section titled “Configuration”Access exceptions
Section titled “Access exceptions”On the Enforcement page, the Access exceptions card (“What verified bots are for”) lists bot behaviors you can tick: Search, Assistants acting for a person, AI training crawlers, Link previews, Feed readers, Uptime and analytics, SEO tools, Security scanners, and Everything else verified. Ticked behaviors let verified bots doing that thing through the clearance gate on enforced routes. Exceptions apply to the organization, or to one site when set for that site.
Unticking a behavior challenges those crawlers on enforced routes. What a given site’s edge validator can verify varies, and the card says so for each site. There is no custom user-agent allowlist: your own bots, partners, and jobs authenticate with per-client machine service tokens minted on the same page (revocable individually, no shared blast radius).
(The WordPress plugin has its own local Good Bots settings. See the plugin guide.)
Verification Methods
Section titled “Verification Methods”For critical applications, verify bot identity:
-
Reverse DNS (recommended for Google/Bing)
Terminal window host <ip_address># Should resolve to *.googlebot.com or *.search.msn.com -
IP Range Verification
- Google publishes Googlebot IP ranges
- Bing publishes bingbot IP ranges
Bot Identification in Detections
Section titled “Bot Identification in Detections”A recognized bot is labeled on the detection with its name and a category, and the detection shows whether its identity verified.
Bot Categories
Section titled “Bot Categories”These are the customer-facing detection classification categories from the agent registry (pkg/agents):
| Category | Examples |
|---|---|
search_crawler |
Googlebot, Bingbot |
training_crawler |
GPTBot, ClaudeBot, CCBot |
ai_search_crawler |
PerplexityBot |
ai_agent / ai_assistant |
ChatGPT-User |
fetcher |
Social link-preview fetchers |
monitoring |
Uptime and monitoring services |
seo_crawler |
Semrush, Ahrefs |
feed_reader |
Feedly, Feedbin |
archiver |
Archive.org |
security_scanner, generic_scraper, commercial_scraper, headless_browser |
Tools and scrapers |
none |
Unrecognized |