Which AI Crawlers to Allow: OpenAI, Anthropic, Perplexity and Google

"Should we block AI?" is the wrong question, because the big vendors no longer run one crawler. They run several, each with a stated purpose, and a robots.txt rule for one does not touch the others. The right question is which of three jobs you want to permit: appearing in answers, being fetched when a user asks, and feeding model training.

Marketing and web leads who need a defensible answer and a file to edit today are the audience. The guide sets out what each vendor says about its bots, suggests a robots.txt layout and shows how to verify what actually visits. Names change often, so recheck the linked pages before touching production.

Three jobs, three kinds of bot

Search bots build the index behind an assistant's answers. Allowing them is what makes your pages eligible to be shown. User-triggered fetchers visit a page because a person asked an assistant about it. Training crawlers collect content that may be used to train future models.

For a B2B brand that wants visibility, the sensible default is to allow the first two and make a deliberate, written choice on the third. Per the vendors' own documents, blocking a training bot leaves you in that vendor's search product. One wrinkle: user-triggered fetchers often skip robots.txt, as the sections below explain.

What each vendor says

What OpenAI documents

Four OpenAI agents are documented. OAI-SearchBot puts sites into ChatGPT search, and a site that opts out of it will not be shown in ChatGPT search answers, although it can still appear as a plain navigational link. GPTBot crawls content that may train foundation models, and disallowing it indicates the content should stay out of training. ChatGPT-User covers actions a person starts, where robots.txt rules may not apply. OAI-AdsBot only checks pages submitted as ads. The settings are independent of one another, and OpenAI says it can take roughly a day for a robots.txt change to register.

How Anthropic splits its bots

Anthropic names three bots. ClaudeBot gathers content for training, Claude-User fetches pages when a person's question needs them, and Claude-SearchBot indexes content to improve search results. Blocking either of the last two may reduce your visibility in user search results. According to Anthropic, its bots respect standard robots.txt directives and the unofficial Crawl-delay extension, and it cautions against blocking by IP address, which may not give a dependable opt-out.

Perplexity's two fetchers

Perplexity describes PerplexityBot as a crawler that helps sites appear in Perplexity search results and says that it is not used to gather content for foundation models, so it advises allowing it. Perplexity-User serves actions a person starts, and it generally disregards robots.txt since a human triggered the visit. Expect up to a day before changes show.

Google's token and fetchers

Google-Extended is a standalone product token that governs whether crawled content may be used for future Gemini training and for grounding. Google says it has no effect on a site's inclusion in Google Search and is no ranking signal, and that crawling still uses the existing Google user agents (the page was updated in July 2026). AI Overviews and AI Mode follow Search rules: a page must be indexed and eligible to appear with a snippet, with no additional requirements. Google also lists user-triggered fetchers, among them Google-Agent and Gemini Notebook, which generally ignore robots.txt because a user requested the fetch. A control to opt out of Search generative features specifically has been described by Google as under exploration, not shipped.

A robots.txt layout to adapt

The sketch below groups bots by job and leaves training crawlers blocked as one possible policy. Treat it as a starting layout, check every name against the vendor pages, and note that a rule naming a specific bot takes precedence over a general rule for all bots.

# Training crawlers: policy choice, blocked here
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

# Search and retrieval: allowed so pages can be shown
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# Everyone else keeps the normal rules
User-agent: *
Disallow: /private/

Date the change in your policy note and test it on a staging path first. The AI content policy template has a section for the decision.

Our call: guard the search bots, not the training question

Debates about GPTBot attract the attention, yet the costlier mistake is usually accidental. A blanket "block all bots" rule, a CDN setting that challenges unknown user agents or a stale wildcard can shut out the search bots you wanted without anyone deciding to. Because search and training bots are separate, deciding on training is a policy matter, not a visibility lever.

We would treat the training choice as a business decision recorded with a date and an owner. It is context, not a reason to touch search access, that Cloudflare's data shows training accounting for about 80% of AI crawling over twelve months, search about 18% and user actions about 2%, and that Anthropic made roughly 38,000 crawls for every referred visit in July 2025. That figure is more than a year old. Everything above shifts the day a vendor merges its search and training crawlers into one agent, because blocking training would then cost visibility. Until a vendor does, keep them apart in your rules and re-read each vendor page every quarter.

Choosing by goal

If you want the widest presence in AI answers, allow every search and user-facing bot and leave training as a separate decision. If you want presence without contributing to future models, allow the search and retrieval bots and disallow the training tokens, accepting that user-triggered fetchers may still visit. If part of your site is gated or licensed, expose only public pages to AI bots and disallow them on paid paths. Whatever you choose, the reason belongs in a short dated note with a named owner.

What to write in the policy note

Keep it to five lines: which bots are allowed and why, which are blocked and why, who decided, the date, and the date of the next review. When a colleague asks in six months why a bot is blocked, that note saves an hour of archaeology. It also makes quarterly review quick, because you can compare the vendor pages against a written baseline instead of memory.

A quarterly routine that stays short

Once a quarter, open the four vendor pages listed above and compare bot names and descriptions with your file. Check the logs for new agents that are not in your rules. Confirm the search bots you wanted are actually reaching key pages and getting normal responses. If something new appears, decide by the three-jobs test, update the note and move on. Thirty minutes is usually enough, and it prevents the slow drift that leaves a rule pointing at a bot that no longer exists.

What a rule cannot promise

Access is a precondition, not a ranking factor: content, structure and reputation decide whether you are cited. Other assistants, such as Copilot and smaller products, have their own rules, so check before assuming. And licensing or contractual limits on content are questions for a qualified professional, not for this page.

Check the logs, because robots.txt is a request

A file asks bots to behave. It does not enforce anything. Look in your server logs for the agents named above and for impostors, using the guides on reading server logs and fake and spoofed crawlers. For the wider visibility plan, see the GEO guide, and for the file some people expect to replace robots.txt, the llms.txt explainer.