Create Crawl Rules
Generate user-agent, allow, and disallow directives for simple website indexing rules before publishing a robots.txt file.
Robots.txt Generator
Generate a clean robots.txt file for search crawlers with user-agent rules, allow paths, disallow paths, crawl delay, and sitemap URL.
robots.txt generatorrobots txt generatorgenerate robots.txtrobots.txt file generator
Browser basedNo signupCopy-ready output
Enter crawl rules
Output
Generated output will appear here.
Add a Sitemap
Include a sitemap URL so search engines can discover the canonical XML sitemap for your site.
Copy Ready Text
The generated output is plain robots.txt content that can be copied into a public file at the root of your website.
Frequently Asked Questions
Where should robots.txt be placed?
Place it at the root of your site, such as https://example.com/robots.txt, so crawlers can find it.
Does robots.txt guarantee pages are not indexed?
No. robots.txt controls crawling, not guaranteed indexing. Use noindex directives when you need stronger indexing control.