A robots.txt generator that builds your crawl directives — user-agent, allow, disallow, crawl-delay, and sitemap URL — with one-click copy. Place the generated file at the root of your domain (e.g. example.com/robots.txt) to control how search engine crawlers access your site.
What is robots.txt?
robots.txt is a plain text file placed at the root of your domain that tells search engine crawlers which parts of your site they can and cannot access. It is the first file a crawler requests when it visits your site. It is a directive, not a rule — well-behaved crawlers like Googlebot respect it, but malicious crawlers ignore it. Do not use robots.txt to hide sensitive content.
robots.txt syntax
A robots.txt file consists of one or more groups. Each group starts with a User-agent line (which crawler the rules apply to) followed by Allow and Disallow lines (which paths). A special Sitemap line at the end points crawlers to your XML sitemap. The wildcard * matches all crawlers. Paths can use wildcards (* matches any sequence, $ matches end of path).
Common robots.txt patterns
Allow everything (default): User-agent: * with no Disallow. Block a private section: Disallow: /admin/. Block search results pages: Disallow: /search/. Block parameter URLs: Disallow: /*?. Block everything: Disallow: /. Always include your sitemap URL so crawlers can discover all your pages efficiently.
When NOT to use robots.txt
Do not use robots.txt to block pages you want kept out of search results — use the noindex meta tag or HTTP headers instead. robots.txt prevents crawling, not indexing. Google can still index a URL it cannot crawl (e.g. if it is linked from other sites). If you Disallow a page in robots.txt, Google will not see your noindex tag because it cannot crawl the page. Use noindex for indexing control, robots.txt for crawl budget control.