AI crawlers explained: GPTBot, ClaudeBot, PerplexityBot and the rest
Your server logs are full of new user agents: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and dozens more. They don't all do the same job. Some collect training data, some index pages for AI search, and some fetch a page live because a user just asked about it. Blocking the wrong one can take you out of AI answers.
Three kinds of AI crawler
Training crawlers collect content that may be used to train future models. Blocking them doesn't affect today's AI search answers.
Search and indexing crawlers build the index an AI assistant searches when it answers. Blocking them removes you from those answers.
User-triggered fetchers load a specific page in real time because a user's conversation needs it.
The crawlers that matter most
User agent names as published by each company:
| Crawler | Company | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training data |
| OAI-SearchBot | OpenAI | Indexing for ChatGPT search |
| ChatGPT-User | OpenAI | Live fetches during a conversation |
| ClaudeBot | Anthropic | Training data |
| Claude-SearchBot | Anthropic | Indexing for Claude's search |
| Claude-User | Anthropic | Live fetches during a conversation |
| PerplexityBot | Perplexity | Indexing for Perplexity answers |
| Perplexity-User | Perplexity | Live fetches during a conversation |
| Google-Extended | robots.txt token controlling Gemini training use; not a separate crawler | |
| Bingbot | Microsoft | Bing index, which grounds Copilot |
| Applebot-Extended | Apple | robots.txt token controlling Apple Intelligence training use |
| CCBot | Common Crawl | Open web archive widely used for training |
A sensible robots.txt default
If you want to appear in AI answers, allow the search and user-triggered crawlers. Whether you allow training crawlers is a business decision; blocking them is reasonable and won't hide you from AI search. Also check your CDN or firewall: many bot-protection settings block AI crawlers before robots.txt is ever read.
What blocking each kind of crawler does
Blocking has a different effect depending on the crawler's job:
| Type | Examples | If you block it |
|---|---|---|
| Training | GPTBot, ClaudeBot, CCBot | Your content isn't collected for future model training. AI search answers are unaffected. |
| Search indexing | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Your pages can drop out of that assistant's search answers and citations. |
| User-triggered | ChatGPT-User, Claude-User, Perplexity-User | The assistant can't open your page when a user asks about it or pastes your link. |
| Opt-out tokens | Google-Extended, Applebot-Extended | Limits training and some AI uses without affecting regular search crawling. |
robots.txt examples
robots.txt groups rules by user agent. A crawler follows the most specific group that names it, so a named group replaces the general “User-agent: *” rules for that bot.
To allow AI search but block training, list the training crawlers with “Disallow: /” and leave the search and user-triggered crawlers allowed:
User-agent: GPTBot, then Disallow: /
User-agent: ClaudeBot, then Disallow: /
User-agent: CCBot, then Disallow: /
User-agent: Google-Extended, then Disallow: /
User-agent: OAI-SearchBot, then Allow: /
User-agent: Claude-SearchBot, then Allow: /
User-agent: PerplexityBot, then Allow: /
Allowing every crawler
To allow everything, a single “User-agent: *” group with “Allow: /” is enough, plus a Sitemap line pointing to your sitemap. Keep private areas such as admin pages or internal search results disallowed for all bots. Test changes with your search engines' robots.txt testers before deploying, since one stray “Disallow: /” can hide your whole site.
How to tell real AI crawlers from fakes
User agent strings are easy to fake, and scrapers often pretend to be well-known bots. Before you trust or block traffic based on the name alone, verify it:
Check the IP address against the ranges the company publishes. OpenAI, for example, publishes IP lists for its crawlers.
For Googlebot and Bingbot, use a reverse DNS lookup on the IP, then a forward lookup to confirm it resolves back to the same address.
Use your CDN's verified-bot signals where available, which already do this matching for known crawlers.
Watch for odd behavior, such as very high request rates or hits on login and checkout pages, which real search crawlers rarely do.
CDN and firewall settings to check
robots.txt is only a request. Your CDN, firewall or hosting provider decides whether a crawler's request is answered at all, and many block AI bots by default or after a single click. If AI crawlers never appear in your logs, start here:
Look for “block AI bots”, “AI scrapers” or bot-fight style settings in your CDN and turn off anything that blocks the search crawlers you want.
Check challenge or CAPTCHA rules. A bot that receives a challenge page instead of your content can't index it.
Review rate limits, which can quietly reject crawlers that request many pages at once.
Make sure any managed robots.txt your CDN adds doesn't contradict the rules you wrote.
After changing settings, request a page with each crawler's user agent and confirm you get your real content back.
Check that crawlers can actually read your pages
Many AI crawlers don't execute JavaScript. If your pricing, features or docs only render client-side, a crawler sees an empty shell. Serve important content as HTML, keep sitemaps current, and consider an llms.txt file that points crawlers to your most important pages.
See which AI bots visit your site
Get Nodded's AI Crawler Analytics tracks visits from 32 AI crawlers, grouped by training, search indexing and live fetching, with a log of every hit. Its AI readiness audit checks access, robots.txt, llms.txt, indexing, rendering and structure, and explains how to fix each issue.
Frequently asked questions
Should I block GPTBot?
Blocking GPTBot only stops your content being used to train OpenAI's models. It doesn't remove you from ChatGPT search, which uses OAI-SearchBot. If you block GPTBot, make sure OAI-SearchBot and ChatGPT-User are still allowed.
Do AI crawlers respect robots.txt?
The major AI companies state that their crawlers follow robots.txt. User-triggered fetchers are treated differently by some providers, since they act on behalf of a person, so check each provider's documentation.
Does blocking ClaudeBot remove me from Claude's answers?
Not by itself. ClaudeBot collects training data, while Claude-SearchBot indexes pages for search and Claude-User fetches pages during conversations. Allow those two if you want to appear when Claude searches the web.
Should I block CCBot?
CCBot builds Common Crawl, a public web archive that many AI models have been trained on. Blocking it limits that use of your content but doesn't affect AI search answers, so it's a business decision rather than a visibility one.
How do I see which AI crawlers visit my site?
Look for their user agents in your server or CDN logs, and verify the IPs, since names can be faked. A crawler analytics tool can group visits by crawler and purpose for you.
Do AI crawlers run JavaScript?
Many don't, or do so only partly. If key content only appears after JavaScript runs, serve it as HTML so every crawler, including AI search bots, can read it.
More in GEO Guides
What is generative engine optimization (GEO)? A practical guide for 2026
llms.txt: what it is, whether you need one, and how to write it
