Robots.txt Generator
Build a correct robots.txt with common presets.
No custom rules yet. Add one, or start from a preset above.
User-agent: *
Allow: /What is robots.txt for?
robots.txt is a plain-text file at the root of your domain (example.com/robots.txt) that tells well-behaved crawlers — search engines, AI training bots, SEO tools — which parts of your site they may fetch. It works on the honor system: it doesn't password-protect anything, and it doesn't remove pages already indexed. It's a request, not a lock.
Rules are grouped by User-agent. A bot that matches a specific group (like GPTBot) follows only that group's rules and ignores the User-agent: * group entirely — the two don't combine.
How to build your robots.txt
- 1Pick a default policy for all crawlers (allow or block), or start from a preset.
- 2Add specific rules to block individual paths or crawlers on top of the default.
- 3Set an optional crawl-delay and point to your sitemap URL.
- 4Fix any warnings, then copy or download the file and upload it to your site's root as /robots.txt.
Common mistakes to avoid
The most damaging mistake is accidentally blocking everything with Disallow: / under User-agent: * — this generator warns you when that happens. Also watch for paths missing a leading slash (admin instead of /admin/), which robots.txt parsers treat as invalid and ignore. Remember that blocking a page in robots.txt only stops crawling — if you need a page fully out of search results, use a noindex meta tag instead (see our meta tag generator), since a blocked page can still appear in results without a snippet if other sites link to it.
Frequently asked questions
›Where does robots.txt need to live?
Exactly at the root of your domain — https://example.com/robots.txt. A file at /blog/robots.txt or on a different subdomain has no effect on that domain; each subdomain needs its own.
›Can robots.txt stop my page from appearing in Google?
Not reliably. It stops Google from crawling the page's content, but the URL can still be indexed (often without a description) if other pages link to it. To fully exclude a page from search results, allow crawling and add a noindex meta tag or HTTP header instead.
›Should I block AI crawlers like GPTBot or CCBot?
That's a judgment call. Blocking them stops your content being used to train some AI models, but it's also opt-out on the honor system — not every crawler respects it, and blocking doesn't affect your search engine visibility since Googlebot and GPTBot are separate crawlers with separate rules.
›What does the sitemap line in robots.txt actually do?
It's a hint that points crawlers to your sitemap.xml so they can discover your pages faster, especially useful for large or frequently updated sites. It's independent of the allow/disallow rules — even a fully blocked site can still declare a sitemap, though there'd be little reason to.
›Do I need a robots.txt file at all?
No — if it's missing, crawlers assume everything is allowed. It's only necessary once you want to restrict crawling of specific paths (like admin panels, search results, or staging areas) or point crawlers to your sitemap.