How to Block AI Crawlers with robots.txt and Cloudflare
Paste a robots.txt block that asks GPTBot and ClaudeBot to stay out, then learn why that file alone cannot enforce it.
The robots.txt Lines That Block AI Crawlers
Add a User-agent group with Disallow: / for each artificial intelligence (AI) crawler you want to block. Then back the file with Cloudflare's AI bot policies or a server rule, because robots.txt is a request and not an access control.
The block below asks the AI training crawlers it names to stay off your pages. It gives each token its own group, and each group ends with Disallow: /. It does not block Googlebot or Applebot, so your site stays in Google and Apple search.
Paste these lines into a plain text file named robots.txt.
# Block AI training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Amazonbot
Disallow: /
# Optional: also block AI search and user-triggered fetchers.
# Doing so removes your site from ChatGPT search, Claude search and Perplexity answers.
# ChatGPT-User and Perplexity-User may not obey these lines (per OpenAI and Perplexity).
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
robots.txt lives at the root of the domain, such as example.com/robots.txt. Anthropic says its opt-out has to be added for each subdomain you want to cover.
OpenAI, Perplexity, Meta and Amazon say their user-triggered fetchers may not follow robots.txt.
What Each AI Crawler Does
Read the last column before you paste the optional lines, because blocking the search tokens removes your site from answers.
| Token | Company | What the company says it is used for | What blocking it costs you |
|---|---|---|---|
| GPTBot | OpenAI | Crawls content that may be used to train OpenAI's generative AI foundation models | OpenAI doesn't say. OpenAI says blocking signals that content should not be used for training. |
| ClaudeBot | Anthropic | Collects web content that could contribute to training Anthropic's generative AI models | Anthropic doesn't say. Blocking signals that future material should be excluded from training. |
| Google-Extended | Controls whether crawled content may train Gemini models and be used for grounding | Google says it does not affect inclusion in Google Search or ranking. | |
| Applebot-Extended | Apple | Lets publishers opt out of their content training Apple's foundation models | Apple doesn't say. Pages can still appear in Apple search results. |
| CCBot | Common Crawl | Crawls for Common Crawl, a non-profit repository of open web crawl data | Common Crawl doesn't say. |
| meta-externalagent | Meta | Crawls for uses including training AI models and improving products by indexing content | Meta doesn't say. |
| Bytespider | ByteDance | ByteDance doesn't say | ByteDance doesn't say. Third-party reports conflict on whether it follows robots.txt. |
| Amazonbot | Amazon | Improves Amazon's products and services and may be used to train Amazon AI models | Amazon doesn't say. |
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT's search features | OpenAI says opted-out sites will not be shown in ChatGPT search answers, though they can still appear as navigational links. |
| ChatGPT-User | OpenAI | Handles actions users start in ChatGPT and Custom GPTs, and is not used for automatic crawling | OpenAI says robots.txt rules may not apply. |
| Claude-SearchBot | Anthropic | Crawls to improve search result quality for Claude users | Stops Claude indexing your content for search. |
| Claude-User | Anthropic | Used when a Claude user asks a question and Claude fetches a website | Stops Claude retrieving your content for user queries. |
| PerplexityBot | Perplexity | Surfaces and links websites in Perplexity search results. Perplexity says it is not used to crawl content for AI foundation models. | Removes your site from Perplexity answers. |
| Perplexity-User | Perplexity | Fetches pages when a user asks a question | Perplexity says this fetcher generally ignores robots.txt rules. |
Training Crawlers vs Search and User Fetchers
The block splits into three groups because each group does a different job. Training tokens gather content for model training. Search tokens decide whether your site appears in AI search answers. User-triggered tokens fetch a page when a person asks for it.
Training tokens signal an opt-out. OpenAI says disallowing GPTBot indicates that a site's content should not be used to train its foundation models. OpenAI and Anthropic do not state a cost for blocking GPTBot or ClaudeBot.
Search tokens cost you visibility. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. Anthropic says disabling Claude-SearchBot stops Claude indexing your content for search.
Timing matters too. OpenAI says a robots.txt update can take about 24 hours to reach its search systems. Meta says its robots.txt changes may take up to 24 hours to apply. Amazon says it uses a cached robots.txt for up to 30 days.
User-triggered fetchers work differently. OpenAI says robots.txt rules may not apply to ChatGPT-User, because users start those actions. Perplexity says Perplexity-User "generally ignores robots.txt rules," because a user requested the fetch.
Meta says Meta-ExternalFetcher "may bypass robots.txt" for fetches a user requested. Amazon says Amzn-User "may not follow all robots.txt directives."
The block above lists neither Meta-ExternalFetcher nor Amzn-User. To refuse them, add a server or CDN rule that matches their User-Agent header.
Google-Extended and Applebot-Extended Do Not Remove You From Search
Google-Extended does not remove your site from Google Search or AI Overviews. Google says the token "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."
Google describes Google-Extended as a standalone product token with no separate crawler. Crawling uses Google's existing user-agent strings, and the token works only as a robots.txt control. Google says it controls whether crawled content may train the Gemini models behind Gemini Apps and the Vertex AI API for Gemini. It also controls use for grounding.
For AI Overviews and AI Mode, Google names different controls. They are the Googlebot robots.txt rules, plus nosnippet, data-nosnippet, max-snippet or noindex. Google's AI features page does not list Google-Extended as a control for AI Overviews.
For the reader side, see our guide to turning off AI Overviews.
Applebot-Extended lets publishers opt out of having their content used to train Apple's foundation models. Those models power generative AI features across Apple products. Apple says, "Applebot-Extended does not crawl webpages." It only decides how data crawled by Applebot is used.
Pages that disallow Applebot-Extended can still appear in Apple search results. Apple's documentation says the rule is not used in Search ranking.
Backing Up robots.txt with Cloudflare and Server Rules
robots.txt is voluntary, and Cloudflare's own docs say some crawlers may ignore it. The Robots Exclusion Protocol appears as RFC 9309, an Internet Engineering Task Force (IETF) Standards Track document published in September 2022. It states: "These rules are not a form of access authorization."
Cloudflare said on July 1, 2025 that it had become the first internet infrastructure provider to block AI crawlers by default. The default applied to new domains, and every new domain is asked at sign-up whether to allow AI crawlers. Cloudflare launched its one-click option to block AI crawlers in September 2024, and more than one million customers had chosen it by July 2025.
Cloudflare's current control sits under Security Settings > "Configure AI bot policies." It has separate Search, Agent and Training policies. Each one can be set to "Block (on all pages)," "Block on pages with ads" or "Allow (do not block)."
The older "Block AI bots" toggle is marked "Deprecating on September 15, 2026." Per Cloudflare's docs, its defaults from September 15, 2026 block Training and Agent bots on pages with ads. Search stays allowed.
AI Crawl Control was formerly called AI Audit, and it is available on all plans. It shows which AI services access your content and sets allow or block rules per crawler. It also tracks which crawlers follow robots.txt.
Managed robots.txt is also available on all plans. Cloudflare prepends its directives to your existing file, or creates one if none exists. The directives include Content-signal: search=yes, ai-train=no, use=reference. They also disallow Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent.
Pay per crawl, which charges AI crawlers for access, is listed as in private beta.
A server rule works at a different layer. The web server refuses requests whose User-Agent header matches a crawler name and returns a 403 response. The User-Agent header can be spoofed, so this stops crawlers that announce themselves and not ones that disguise themselves.
Common Crawl warns that some crawlers falsely identify themselves as CCBot. It says the real CCBot uses dedicated IP ranges with reverse domain name system (DNS) records.
Anthropic recommends robots.txt over IP blocking, because IP blocking can stop its bots from reading your robots.txt. Anthropic also publishes its crawler IP list on its own site.
The Perplexity Dispute
Cloudflare said Perplexity used undeclared crawlers that impersonated Google Chrome on macOS. In its blog post dated August 4, 2025, Cloudflare said these crawlers used IP addresses outside Perplexity's published ranges.
Cloudflare said the crawlers sometimes ignored robots.txt or did not fetch it, across "tens of thousands of domains and millions of requests per day." It reported 20 to 25 million daily requests from Perplexity's declared crawler and 3 to 6 million from the alleged stealth crawler.
In response, Cloudflare de-listed Perplexity as a verified bot and added blocking heuristics to its managed rules. The same post said OpenAI's ChatGPT-User stopped crawling when disallowed or blocked.
Perplexity replied the same day in its own blog post. It says Cloudflare's post "managed to get almost everything wrong." Perplexity says Cloudflare confused it with 3 to 6 million daily requests of unrelated traffic from BrowserBase, a third-party cloud browser service.
Perplexity also argues that user-driven agents differ from bots and do not store or train on fetched content.
Should You Block AI Crawlers?
Block the training tokens if you object to AI training on your pages. Leave the search tokens open if you want your site to appear in ChatGPT, Claude or Perplexity answers.
The IETF AI Preferences (aipref) working group is drafting a standard vocabulary for AI-use preferences. Its drafts targeted submission by August 31, 2026, and neither is a published RFC according to the working group page.
This block is not for site owners who want AI assistants to surface their pages. For that, leave OAI-SearchBot, Claude-SearchBot and PerplexityBot unblocked. Owners who need a block enforced should not rely on robots.txt alone, because the file is only a request.
We would revisit this advice if the aipref drafts become a published standard, or if OpenAI, Anthropic or Perplexity change what they say about robots.txt.
If your concern is copying of images rather than text, read our guide to AI art theft. For the copyright question, see our explainer on whether AI training is copyright infringement. See also our guide to no-AI policies.
Paste the training block into your robots.txt today, then check each Training policy in Cloudflare's AI bot policies documentation.
Sources
- OpenAI, Overview of OpenAI Crawlers
- Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Google Search Central, Google's common crawlers
- Google Search Central, AI features and your website
- Apple Support, About Applebot
- Common Crawl, CCBot
- Perplexity, Perplexity Crawlers
- Meta for Developers, Meta Web Crawlers
- Amazon, Amazonbot
- IETF, RFC 9309 Robots Exclusion Protocol
- IETF, AI Preferences (aipref) working group
- Cloudflare, press release, July 1, 2025
- Cloudflare blog, Perplexity is using stealth, undeclared crawlers
- Perplexity blog, Agents or bots? Making sense of AI on the open web
- Cloudflare docs, Block AI bots and AI bot policies
- Cloudflare docs, Managed robots.txt
- Cloudflare docs, AI Crawl Control
Frequently asked questions
▸ What is GPTBot?
▸ What is ClaudeBot?
▸ Does GPTBot respect robots.txt?
▸ Should I block AI crawlers?
▸ How to block ChatGPT from website?
▸ Is AI scraping illegal?
Latest related briefings
New AI Safety Guidelines Aim to Shield Kids Online—Will They Work?
IEEE released new AI safety guidelines to protect kids online, but their impact depends on adoption by tech companies and schools. Parents should stay vigilant.
Read analysis JOBS LABORWisconsin’s $6.6M AI Job Grants: What It Means for Workers
Wisconsin’s $6.6M in AI workforce grants could help some workers adapt, but many jobs face risk. Here’s what everyday people need to know.
Read analysis REGULATION POLICYEU AI Act Targets OpenAI, Anthropic: What It Means for You
EU regulators’ first cases against OpenAI and Anthropic may bring new AI protections—and limits—for everyday people, from workers to parents and students.
Read analysis