Generate a correct robots.txt and test it before you upload
A robots.txt file tells well-behaved crawlers which parts of your site they may fetch. The Robots.txt Generator builds one from simple choices — pick a crawler, add Allow and Disallow paths, list your sitemaps — and shows the finished file as you go. A one-click preset adds groups that block the crawlers used to gather AI training data, each described honestly so you know what blocking it actually does.
The built-in tester applies the same precedence rules Google documents (longest matching rule wins, Allow wins a tie, * and $ wildcards) to any path and crawler you type, so you can confirm a rule before it goes live. Everything runs in your browser; nothing is fetched from your site.
How to use the Robots.txt Generator
- Start from the default group for all crawlers and edit its Disallow and Allow paths, or add groups for specific bots such as Googlebot or Bingbot.
- Press Block AI training crawlers to add GPTBot, Google-Extended, ClaudeBot, CCBot, Applebot-Extended and others with
Disallow: /. Remove any you want to keep. - List your sitemap URLs — the Sitemap Generator can make one.
- In the tester, enter a path like
/admin/settingsor a full URL and choose a crawler to see whether it is allowed or blocked and which rule decided. - Copy the file or download
robots.txtand upload it to the root of your domain.
Robots.txt Generator features
Per-crawler groups
Choose from 20 common search, AI and SEO crawlers with a plain description of each, or type any custom token.
Honest AI crawler preset
Explains the difference between training crawlers (GPTBot, ClaudeBot, CCBot), control tokens (Google-Extended, Applebot-Extended) and on-demand fetchers (ChatGPT-User) so you don’t block search visibility by mistake.
Google-accurate tester
Evaluates longest-match precedence, Allow-over-Disallow ties, * and $ wildcards, query strings and user-agent group selection the way Google’s open-source parser does.
Live validation
Flags paths that don’t start with /, invalid sitemap URLs, duplicate groups and Crawl-delay on Googlebot, which ignores it.
Copy or download
Get a clean, UTF-8 robots.txt ready to upload.
How crawlers read robots.txt
The file lives at the root of a host (https://example.com/robots.txt) and only applies to that host and protocol. It is made of groups: one or more User-agent lines followed by Allow and Disallow rules. A crawler uses the group whose token best matches its name; if none matches, it uses the * group. Paths are matched as prefixes, * matches any characters and $ anchors the end of the URL, which includes the query string.
When several rules match a URL, Google uses the most specific one — the rule with the longest path — and if an Allow and a Disallow are equally long, the Allow wins. For example, with Disallow: /admin/ and Allow: /admin/public/, the URL /admin/public/help is allowed. Bing follows similar rules, but older or simpler crawlers may read the file top to bottom instead.
| Pattern | Matches | Does not match |
|---|---|---|
/admin/ | /admin/, /admin/users | /administrator |
/search | /search, /search?q=a, /searching | /s/earch |
/*.pdf$ | /a.pdf, /files/b.pdf | /a.pdf?download=1 |
/*? | any URL with a query string | /page |
Disallow: (empty) | nothing — allows everything | — |
Blocking AI crawlers: what each token does
Companies use different crawlers for different purposes, and blocking the wrong one can cost you visibility. GPTBot, ClaudeBot, CCBot, Bytespider and meta-externalagent collect pages for model training. Google-Extended and Applebot-Extended are not crawlers at all but opt-out tokens: blocking them stops your content being used to train Gemini or Apple models while leaving Google Search and Siri untouched. OAI-SearchBot and PerplexityBot index pages for AI search products, and ChatGPT-User fetches a page only when a user asks about it.
Be aware of the limits: robots.txt is voluntary. Reputable companies honour it, but it is not a legal or technical barrier, and content already crawled is not removed. For enforcement you need server rules such as those from the .htaccess Generator or a firewall.
Common robots.txt mistakes
- Using it to hide pages from search. A blocked URL can still be indexed from links. To keep a page out of results, allow crawling and add a
noindexmeta tag with the Meta Tag Generator. - Blocking CSS and JavaScript. Google renders pages; if it can’t load your assets it may misjudge mobile usability and content.
- Relying on Crawl-delay for Google. Googlebot ignores it. Bing and Yandex support it.
- Case and placement errors. Paths are case-sensitive, and the file must be at the root —
/blog/robots.txtdoes nothing. - Forgetting staging sites. A
Disallow: /copied from staging to production removes the whole site from crawling.
Robots.txt Generator FAQ
Do I need a robots.txt file?
Not strictly — without one, crawlers may fetch everything. But a robots.txt generator makes it easy to keep bots out of admin areas, search result pages and duplicate URLs, and to point them to your sitemap.
Does robots.txt stop a page from appearing in Google?
No. It stops crawling, not indexing. A blocked URL can still show up with no snippet if other pages link to it. Use a noindex meta tag (and let the page be crawled) to remove it from results.
How do I block ChatGPT from using my website?
Add “User-agent: GPTBot” followed by “Disallow: /” to block training. To also stay out of ChatGPT search results, block OAI-SearchBot; to stop on-demand browsing, block ChatGPT-User. The “Block AI training crawlers” button adds the training bots only.
What does Google-Extended do?
It is a control token, not a separate crawler. Disallowing Google-Extended tells Google not to use your content to train Gemini models or for grounding. It has no effect on your pages in Google Search or on AI Overviews.
Does Google respect Crawl-delay?
No. Google ignores the Crawl-delay directive; its crawl rate is adaptive and can be limited in Search Console. Bing and Yandex do honour it, so the generator still lets you set it per group and warns when it is attached to Googlebot.
How does the tester decide allowed or blocked?
It parses the generated file, picks the group whose user-agent token matches the crawler (or the * group), finds every Allow and Disallow rule matching the path with * and $ wildcards, and applies the longest rule — with Allow winning ties — exactly as Google documents.
Where do I upload the file?
In the top-level directory of your web server so it is reachable at https://yourdomain.com/robots.txt. Each subdomain needs its own file.