robots.txt Generator

Create a clean robots.txt for your site. Add user-agent rules, allow/disallow paths, an optional crawl delay and a sitemap URL.

Rules

User-agent(s) Use * for all crawlers, or comma-separated bot names such as Googlebot, Bingbot. Multiple names share the rules below.

Allow paths (one per line)

Disallow paths (one per line)

Add an explicit empty Disallow: (allow all)

Additional groups (optional) Add full robots.txt groups with their own User-agent, Allow, Disallow, Crawl-delay or Sitemap lines. Lines starting with # are comments.

Crawl-delay (seconds, optional)

Sitemap URL (optional)

robots.txt

Remember

robots.txt is a voluntary convention, not an access control mechanism. It does not protect private data.
Rules are matched by path prefix. Disallow: /admin/ blocks that directory, but not /admin without a trailing slash.
Use the most specific Allow rule to reopen a path inside a blocked directory.
Keep the file at the site root: https://example.com/robots.txt.
An empty Disallow: line explicitly allows everything for its user-agent group. Use the checkbox to add it.
Use the additional-groups box for crawlers that need different rules from the main group.