robots.txt AI bot generator
Save on GoogleFree robots.txt generator for AI crawlers. Block or allow GPTBot, ClaudeBot, Google-Extended, PerplexityBot and more, then copy the file. Everything runs in your browser.
Inputs
AI bots
Options
Output
Your robots.txt
robots.txt is a request, not a lock. Crawlers that ignore it can still fetch your pages.
What is a robots.txt AI bot blocker?
A robots.txt file tells crawlers which parts of your site they may visit. This generator writes the User-agent and Disallow lines for popular AI crawlers, split into training crawlers and search or assistant crawlers, so you can block one group and allow the other. Rules are voluntary and only affect crawlers that choose to obey them.
robots.txt AI bot reference: 70 crawlers and directives
Rules follow IETF RFC 9309, the Robots Exclusion Protocol. Tokens I could not confirm in an operator’s documentation are marked in the notes, so verify them before relying on them.
| Crawler / Directive / Query | Operator / Standard Body | Exact robots.txt Rule / Syntax | Scope, Behavior & Crawler Impact | Category |
|---|---|---|---|---|
| GPTBot | OpenAI | User-agent: GPTBot
Disallow: / | Collects public web content that may be used to train OpenAI models. Blocking it does not remove you from ChatGPT search. | AI Training Bot |
| ChatGPT-User | OpenAI | User-agent: ChatGPT-User
Disallow: / | Opens a page when a ChatGPT user asks it to. OpenAI’s docs note robots.txt may not apply to user-initiated fetches, so check them. | AI User Fetcher |
| ClaudeBot | Anthropic | User-agent: ClaudeBot
Disallow: / | Anthropic’s crawler that collects public web content to train Claude models. | AI Training Bot |
| Anthropic-AI | Anthropic | User-agent: anthropic-ai
Disallow: / | Older token that appears in block lists. ClaudeBot is the documented crawler, so I could not confirm this one is still active. | AI Training Bot |
| Google-Extended | User-agent: Google-Extended
Disallow: / | A control token, not a crawler. It stops use of your content for Gemini and Vertex AI and does not affect Search inclusion or ranking. | AI Training Bot | |
| Bytespider | ByteDance | User-agent: Bytespider
Disallow: / | ByteDance’s crawler, reported to collect data for its AI products. Some site owners report it ignores robots.txt, so add firewall rules. | AI Training Bot |
| ByteDance | ByteDance | User-agent: ByteDance
Disallow: / | I could not confirm a documented crawler token with this name. Use Bytespider and check ByteDance’s documentation. | AI Training Bot |
| Applebot-Extended | Apple | User-agent: Applebot-Extended
Disallow: / | A control token, not a crawler. It tells Apple not to use your content to train its models. Applebot search is unaffected. | AI Training Bot |
| Meta-ExternalAgent | Meta | User-agent: Meta-ExternalAgent
Disallow: / | Meta’s crawler for AI model training and other product uses. | AI Training Bot |
| Meta-ExternalFetcher | Meta | User-agent: Meta-ExternalFetcher
Disallow: / | Fetches a URL that a person shares in a Meta product. Meta’s docs say it may bypass robots.txt because the fetch is user-initiated. | AI User Fetcher |
| Amazonbot | Amazon | User-agent: Amazonbot
Disallow: / | Amazon’s crawler, used for services such as Alexa and to improve Amazon AI products. | AI Training Bot |
| Cohere-ai | Cohere | User-agent: cohere-ai
Disallow: / | Token that appears in block lists for Cohere. I could not confirm details, so check Cohere’s documentation. | AI Training Bot |
| Cohere-training | Cohere | User-agent: cohere-training-data-crawler
Disallow: / | Token reported in block lists for Cohere’s training-data crawler. Verify it with Cohere before relying on it. | AI Training Bot |
| MistralAI | Mistral | User-agent: MistralAI-User
Disallow: / | The listed token is MistralAI-User, a user-initiated fetcher. I could not confirm a Mistral training crawler. | AI User Fetcher |
| DeepSeek-Bot | DeepSeek | User-agent: DeepSeekBot
Disallow: / | I could not confirm a documented DeepSeek crawler token. Block lists use DeepSeekBot, so verify before relying on it. | AI Training Bot |
| AI2Bot | Allen Institute for AI | User-agent: AI2Bot
Disallow: / | Crawler listed for the Allen Institute for AI’s research and model-training datasets. | AI Training Bot |
| Omgilibot | Omgili / Webz.io | User-agent: omgilibot
Disallow: / | Collects web text for Omgili and Webz.io datasets. | Open Dataset |
| YouBot | You.com | User-agent: YouBot
Disallow: / | You.com’s crawler for its AI search index and answers. | AI Search Bot |
| PhindBot | Phind | User-agent: PhindBot
Disallow: / | Token listed for Phind’s developer-focused AI search. I could not confirm details, so check Phind’s documentation. | AI Search Bot |
| KagiBot | Kagi | User-agent: Kagibot
Disallow: / | Crawler for Kagi’s search index. Kagi documents the user agent Kagibot. | AI Search Bot |
| Perplexity-Ai | Perplexity | User-agent: PerplexityBot
Disallow: / | I could not confirm a token named Perplexity-Ai. Perplexity documents PerplexityBot, which builds its search index, and Perplexity-User. | AI Search Bot |
| Timpibot | Timpi | User-agent: Timpibot
Disallow: / | Crawler for Timpi’s decentralized search index. | AI Search Bot |
| Claude-Web | Anthropic | User-agent: Claude-Web
Disallow: / | Older Anthropic token. The currently documented tokens are ClaudeBot, Claude-User and Claude-SearchBot, so this one may be inactive. | AI User Fetcher |
| VelenPublicWebCrawler | Velen | User-agent: VelenPublicWebCrawler
Disallow: / | Token that appears in AI-crawler block lists. I could not verify the operator’s documentation, so check it first. | Open Dataset |
| Webzio-Extended | Webz.io | User-agent: Webzio-Extended
Disallow: / | Token reported by Webz.io for opting out of AI use of its data. Verify it with Webz.io. | Open Dataset |
| OAI-SearchBot | OpenAI | User-agent: OAI-SearchBot
Allow: / | Powers search results in ChatGPT. Allow it to appear in ChatGPT search while you block GPTBot for training. | AI Search Bot |
| PerplexityBot | Perplexity AI | User-agent: PerplexityBot
Allow: / | Indexes pages for Perplexity answers and citations. Allowing it keeps your pages eligible to be cited. | AI Search Bot |
| Bingbot | Microsoft | User-agent: Bingbot
Allow: / | Bing’s crawler. Bing results also feed some Microsoft Copilot answers. Blocking it removes you from Bing Search. | Search Engine |
| Googlebot | User-agent: Googlebot
Allow: / | Core Google Search crawler. Allow it to stay indexed. Google-Extended does not control it. | Search Engine | |
| DuckDuckBot | DuckDuckGo | User-agent: DuckDuckBot
Allow: / | DuckDuckGo’s crawler for its search results. | Search Engine |
| Applebot | Apple | User-agent: Applebot
Allow: / | Apple’s crawler for Siri, Spotlight and Safari suggestions. Applebot-Extended only controls AI training use. | Search Engine |
| Yahoo! Slurp | Yahoo | User-agent: Slurp
Allow: / | Yahoo’s search crawler. The robots.txt token is Slurp. | Search Engine |
| YandexBot | Yandex | User-agent: Yandex
Allow: / | Yandex’s search crawler. Yandex documents the robots.txt token Yandex. | Search Engine |
| Baiduspider | Baidu | User-agent: Baiduspider
Allow: / | Baidu’s search crawler. | Search Engine |
| Naver Yeti | Naver | User-agent: Yeti
Allow: / | Naver’s search crawler. The robots.txt token is Yeti. | Search Engine |
| Exabot | Exalead | User-agent: Exabot
Allow: / | Crawler for the Exalead search engine. | Search Engine |
| Sogou web spider | Sogou | User-agent: Sogou web spider
Allow: / | Sogou’s search crawler. | Search Engine |
| Qwantify | Qwant | User-agent: Qwantify
Allow: / | Crawler for Qwant, a European privacy-focused search engine. | Search Engine |
| MojeekBot | Mojeek | User-agent: MojeekBot
Allow: / | Crawler for the independent Mojeek search index. | Search Engine |
| PetalBot | Huawei | User-agent: PetalBot
Allow: / | Huawei’s crawler for Petal Search. | Search Engine |
| CCBot | Common Crawl | User-agent: CCBot
Disallow: / | Builds the open Common Crawl dataset that many AI models use for training. Blocking stops future collection, not past copies. | Open Dataset |
| Diffbot | Diffbot | User-agent: Diffbot
Disallow: / | Extracts structured data for Diffbot’s Knowledge Graph and AI products. | Open Dataset |
| Scrapy | Scrapy framework | User-agent: Scrapy
Disallow: / | Default user agent of Scrapy projects. It obeys robots.txt only when the operator turns on ROBOTSTXT_OBEY. | Scraper / Client |
| BeautifulSoup | Python library | User-agent: BeautifulSoup
Disallow: / | A parsing library, not a crawler. It sends no user agent of its own, so this rule does nothing. Scripts often send python-requests. | Scraper / Client |
| Apache-HttpClient | Apache (Java) | User-agent: Apache-HttpClient
Disallow: / | Default user agent of Java HTTP clients. Scripts using it usually ignore robots.txt unless the author adds support. | Scraper / Client |
| Wget | GNU | User-agent: Wget
Disallow: / | Command-line downloader. It honors robots.txt in recursive mode unless the user turns that off. | Scraper / Client |
| curl | cURL project | User-agent: curl
Disallow: / | Command-line client. It does not read robots.txt, so this rule has no effect. Use server rules instead. | Scraper / Client |
| GitHub-Copilot | GitHub / Microsoft | User-agent: GitHub-Copilot
Disallow: / | I could not confirm a documented crawler token with this name. Do not rely on this rule. | Scraper / Client |
| Cursor-Agent | Cursor | User-agent: Cursor-Agent
Disallow: / | I could not confirm a documented crawler token with this name. Check Cursor’s documentation. | Scraper / Client |
| Sourcegraph-Bot | Sourcegraph | User-agent: Sourcegraph-Bot
Disallow: / | I could not confirm a documented crawler token with this name. Check Sourcegraph’s documentation. | Scraper / Client |
| SemanticScholarBot | Allen Institute for AI | User-agent: SemanticScholarBot
Disallow: / | Crawler for Semantic Scholar’s academic paper index. | Research / Archive |
| InternetArchiveBot | Internet Archive | User-agent: InternetArchiveBot
Disallow: / | Repairs dead links on Wikipedia with archived copies. It is not a general web crawler, so blocking it rarely matters. | Research / Archive |
| Archive.org_bot | Internet Archive | User-agent: archive.org_bot
Disallow: / | Internet Archive’s crawler for the Wayback Machine. | Research / Archive |
| turnitinbot | Turnitin | User-agent: turnitinbot
Disallow: / | Collects pages to compare against student work for plagiarism detection. | Research / Archive |
| ia_archiver | Alexa / Internet Archive | User-agent: ia_archiver
Disallow: / | Legacy Alexa crawler token that still appears in robots.txt files. Activity may be low now. | Research / Archive |
| Disallow entire site | IETF RFC 9309 | User-agent: *
Disallow: / | Blocks every compliant crawler from all paths, including search engines. Use with care. | Syntax Core |
| Allow entire site | IETF RFC 9309 | User-agent: *
Disallow: | An empty Disallow value allows all paths. | Syntax Core |
| Block a specific directory | IETF RFC 9309 | User-agent: *
Disallow: /admin/ | Blocks the folder and everything under it. | Syntax Core |
| Allow a subdirectory inside a disallowed root | IETF RFC 9309 | User-agent: *
Disallow: /private/
Allow: /private/public/ | The longer, more specific rule wins, so the subfolder stays crawlable. | Syntax Core |
| End-of-URL matching with $ | IETF RFC 9309 | User-agent: *
Disallow: /*.pdf$ | The $ anchors the end of the URL, so only URLs ending in .pdf match. | Syntax Core |
| Wildcard mid-path matching with * | IETF RFC 9309 | User-agent: *
Disallow: /temp/*/draft | The * matches any characters. A trailing * is redundant because rules already match by prefix. | Syntax Core |
| Query parameter blocking | IETF RFC 9309 | User-agent: *
Disallow: /*?* | Blocks URLs with a query string, which stops faceted-search crawling. It also blocks parameter pages you may want indexed. | Syntax Core |
| Allow precedence over Disallow | IETF RFC 9309 | User-agent: *
Disallow: /folder/
Allow: /folder/page.html | The most specific (longest) matching rule wins. On a tie, Allow wins. | Syntax Core |
| Declaring an XML sitemap | Sitemaps protocol | Sitemap: https://example.com/sitemap.xml | Use a full URL. It can appear anywhere, applies to all crawlers, and you can list several. | Syntax Core |
| Allowing llms.txt while blocking AI bots | Convention | User-agent: GPTBot
Allow: /llms.txt
Disallow: / | The longer Allow rule beats Disallow: /, so only llms.txt stays readable. Repeat it in each blocked group. | Edge Case |
| Crawl-delay directive | Non-standard | User-agent: Bingbot
Crawl-delay: 10 | Not part of RFC 9309. Bing documents support for it and Google ignores it. Support varies by crawler. | Edge Case |
| Case sensitivity in URLs | IETF RFC 9309 | User-agent: *
Disallow: /Private | Paths are case sensitive, so /private is not blocked by this rule. User-agent names are not case sensitive. | Edge Case |
| Root location requirement | IETF RFC 9309 | https://example.com/robots.txt | It must sit at the root of the host. It covers only that protocol, host and port, so each subdomain needs its own file. | Edge Case |
| HTTP 404 vs 403 vs 500 | IETF RFC 9309 | HTTP 404 / 403 / 500 | 4xx codes are treated as no robots.txt, so everything is crawlable. 5xx errors make crawlers assume a full disallow for a time. | Edge Case |
| Maximum robots.txt file size | IETF RFC 9309 | Parsed size limit: 500 KiB | Crawlers must parse at least 500 KiB. Google ignores anything past that limit. | Edge Case |
robots.txt AI bot FAQ
| Question | Answer |
|---|---|
| What is robots.txt and how does it work? | robots.txt is a plain text file at the root of a site. Crawlers fetch it before crawling and follow the User-agent, Allow and Disallow rules for their name. It is a voluntary convention defined in RFC 9309, so compliant crawlers obey it and others may not. |
| How do I block all AI training crawlers with robots.txt? | Add a User-agent line for each training crawler, such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Meta-ExternalAgent, followed by Disallow: /. The generator above writes these lines for you. |
| What is the difference between GPTBot and OAI-SearchBot? | According to OpenAI’s documentation, GPTBot collects content that may be used to train models, while OAI-SearchBot supports search results in ChatGPT. Block GPTBot and allow OAI-SearchBot to opt out of training but stay in ChatGPT search. |
| Does blocking Google-Extended hurt my Google Search rankings? | No. Google says Google-Extended controls whether your content is used for Gemini AI training and grounding. It does not affect inclusion in Google Search or ranking. |
| How do I block ClaudeBot from scraping my website? | Add User-agent: ClaudeBot followed by Disallow: / to your robots.txt file. Anthropic documents ClaudeBot as its training crawler and states that its crawlers follow robots.txt. |
| Can AI bots ignore robots.txt directives? | Yes. robots.txt is voluntary. Major companies say their crawlers respect it, but some scrapers and user-initiated fetchers do not. Use firewall or CDN bot rules for a hard block. |
| Where must the robots.txt file be hosted? | At the root of the host, for example https://example.com/robots.txt. Each subdomain and protocol needs its own file. |
| How do I allow llms.txt while disallowing AI crawlers in robots.txt? | In each blocked group, add Allow: /llms.txt before Disallow: /. The longer, more specific Allow rule wins over Disallow: /, so only the llms.txt file stays readable. |