Build a valid robots.txt to tell search engines which paths to crawl or avoid, set a crawl delay and point to your sitemap. Copy it to your site's root. Generated in your browser.
Robots.txt Generator helps you optimise and publish with confidence, producing clean, standards-friendly output you can paste straight into your site.
No — it asks crawlers not to index it; use auth to truly protect pages.
At your domain root, e.g. example.com/robots.txt.
No — local only.
User-agent: *
Disallow: /admin/
Disallow: /cart
Sitemap: https://example.com/sitemap.xml