GetOmniOne

robots.txt Generator

Generate a valid robots.txt file.

Rules
robots.txt

About the robots.txt Generator

Generate a valid robots.txt file with allow/disallow rules, crawl-delay and sitemap references. Tell search engines what to crawl without hand-writing the syntax.

How to use it

  1. 1Add the paths you want to allow or disallow.
  2. 2Optionally set a crawl-delay and your sitemap URL.
  3. 3Copy the generated robots.txt to your site root.

Frequently asked questions

Does robots.txt keep a page private?
No — it only requests that compliant crawlers skip a path. Use authentication for anything that must stay private.

Related tools

Meta Tag GeneratorBuild SEO and Open Graph meta tags.Slug GeneratorTurn any title into a clean URL slug.URL Encoder & DecoderPercent-encode and decode URLs.