Robots.txt Generator and Tester
A robots.txt file tells crawlers which paths of your site they may fetch, using User-agent groups with Allow and Disallow rules and Sitemap lines. Pick a preset for Blogger or WordPress, choose which of 12 AI crawlers to block, then test any URL: the tester applies the same longest-match rules Google uses. The Blogger example below blocks /search/label/SEO for Googlebot but keeps it open to AdSense.
Advertisement
How to make and test a robots.txt file
- Choose a preset and type your site address. Blogger and WordPress presets match each platform's own defaults.
- Adjust the rules. Each row is one Allow or Disallow line for a user agent; rows with the same agent become one group.
- Pick AI crawlers to block. "Block AI training only" stops training crawlers but keeps you visible in ChatGPT, Claude and Perplexity search.
- Read the audit and test your important URLs: a post, a label page, an image, your CSS. Fix anything blocked by mistake.
- Publish it at the root of your domain (not in a folder), then check it in Google Search Console's robots.txt report.
AI crawler user agents
Each company publishes the user agent tokens its crawlers obey. Training crawlers collect pages for model training; search crawlers index pages so they can be cited in AI answers; user-requested fetchers load a page when someone asks the assistant to read it, and some of them do not follow robots.txt.
| User agent | Company | Type | What it does |
|---|---|---|---|
GPTBot | OpenAI | AI training | Collects pages that may be used to train OpenAI models. |
OAI-SearchBot | OpenAI | AI search | Indexes pages so they can appear as sources in ChatGPT search. |
ChatGPT-User | OpenAI | User-requested fetch | Fetches a page when a ChatGPT user asks for it; may not follow robots.txt. |
ClaudeBot | Anthropic | AI training | Collects pages that may be used to train Claude models. |
Claude-SearchBot | Anthropic | AI search | Indexes pages for search results in Claude. |
Claude-User | Anthropic | User-requested fetch | Fetches a page when a Claude user asks for it. |
Google-Extended | AI training | Control token, not a crawler: stops Googlebot-crawled pages being used for Gemini training and grounding. No effect on Google Search. | |
Applebot-Extended | Apple | AI training | Control token: stops Applebot-crawled pages being used to train Apple models. Applebot search crawling continues. |
PerplexityBot | Perplexity | AI search | Indexes pages for Perplexity answers. |
Perplexity-User | Perplexity | User-requested fetch | Fetches a page when a Perplexity user asks for it; may not follow robots.txt. |
meta-externalagent | Meta | AI training | Collects pages for Meta AI training and products. |
CCBot | Common Crawl | AI training | Builds the open Common Crawl dataset that many AI models are trained on. |
Blocking AI search crawlers has a cost. OAI-SearchBot, Claude-SearchBot and PerplexityBot are how pages get cited in AI answers. If you want AI visibility but not AI training, use "Block AI training only".
Advertisement
How crawlers read robots.txt
These are the rules the tester applies, from RFC 9309 and Google's documentation. Every example is checked by the tool's own tests.
| Rules | URL | Result |
|---|---|---|
User-agent: * | /page | Allowed (Allow: /p) |
User-agent: * | /folder/page | Allowed (Allow: /folder) |
User-agent: * | /page.htm | Blocked (Disallow: /*.htm) |
User-agent: * | /Fish.asp | Allowed |
User-agent: * | /index.php?lang=en | Allowed |
User-agent: googlebot | /private/a.jpg (Googlebot-Image) | Blocked (Disallow: /private/) |
User-agent: * | /robots.txt | Allowed (always) |
- One group per crawler. A crawler uses the group with the most specific matching user agent (Googlebot-Image falls back to Googlebot) and ignores the
*group if a specific one exists. Groups for the same agent are merged. - Longest rule wins, whatever the order. On a tie, Allow wins.
- Paths are case-sensitive; user agent names are not.
- Limits: Google reads the first 500 KiB, caches the file for up to 24 hours and ignores Crawl-delay and Noindex lines.
Where to put robots.txt
Blogger
Settings → Crawlers and indexing → Enable custom robots.txt, then paste the file. Keep custom robots header tags off unless you know you need them. Blogger serves it at /robots.txt on your blog's domain.
WordPress
WordPress serves a virtual robots.txt until a real file exists. Upload robots.txt to the site root by FTP or the host's file manager, or use your SEO plugin's editor (Yoast: Tools → File editor; Rank Math: General Settings → Edit robots.txt).
Disclaimer: robots.txt is a request that reputable crawlers follow; it does not secure private content. Crawler names and behaviour change; this page lists tokens published by their operators as of 17 October 2026.
Sources
Frequently asked questions
What is a robots.txt file?
Robots.txt is a plain text file at the root of a site, such as https://www.example.com/robots.txt, that tells crawlers which paths they may fetch. Each group starts with one or more User-agent lines followed by Allow and Disallow rules, and Sitemap lines can point to your sitemaps. It is a public request, not a lock: well-behaved crawlers follow it, but it does not hide or protect pages.
What is the best custom robots.txt for Blogger?
Use Blogger's default plus your pages sitemap: allow Mediapartners-Google (AdSense) everywhere, disallow /search for everyone else so label and search pages are not crawled, allow /, and list /sitemap.xml and /sitemap-pages.xml. Paste it in Settings, Crawlers and indexing, Enable custom robots.txt. Do not disallow / or your posts will drop out of Google.
How do I block ChatGPT and other AI crawlers?
Add a group for each AI user agent with Disallow: /. GPTBot (OpenAI training), ClaudeBot (Anthropic training), CCBot (Common Crawl), meta-externalagent (Meta) and the control tokens Google-Extended and Applebot-Extended cover AI training. OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages for AI search, so blocking them can remove you from AI answers. Tick the crawlers in the tool and it writes the group for you.
Does blocking Google-Extended affect my Google rankings?
No. Google-Extended is a control token, not a separate crawler: blocking it stops content crawled by Googlebot from being used to train Gemini models and for grounding, but Google states it does not affect inclusion or ranking in Google Search. Keep Googlebot allowed.
Does robots.txt remove a page from Google?
No. Disallow stops Google from crawling a page, but the URL can still be indexed without its content if other pages link to it. To remove a page from search, allow crawling and add a noindex meta robots tag or X-Robots-Tag header. Google stopped supporting noindex lines inside robots.txt in 2019.
How do wildcards work in robots.txt?
An asterisk matches any sequence of characters and a dollar sign at the end anchors the end of the URL. Disallow: /*.pdf$ blocks /files/guide.pdf but not /guide.pdf?download=1. When several rules match, the longest rule wins, and when an Allow and a Disallow rule are equally long, Allow wins. Rules are case-sensitive: /Fish and /fish are different paths.
Is there a size limit or crawl-delay in robots.txt?
Google reads only the first 500 KiB of a robots.txt file and ignores the rest. Google ignores Crawl-delay; Bing and Yandex read it. Google generally caches robots.txt for up to 24 hours, so changes can take a day to apply. The audit in this tool warns about all three.
Share this robots.txt generator
Advertisement