Robots.txt generator.
Create a valid robots.txt with the right allow and disallow rules, crawl-delay, and sitemap directives, then copy or download it. Presets for allow-all, block-all, and WordPress.
One plan, $199/month. No contract, cancel anytime.
User-agent: * Sitemap: https://yoursite.com/sitemap.xml
Upload this to the root of your site as /robots.txt. Blocking a page here stops crawling, not indexing, use a noindex meta tag to keep a page out of results.
Robots.txt FAQ
Does robots.txt stop a page from being indexed?
No. Robots.txt controls crawling, not indexing. A blocked URL can still appear in Google if other pages link to it. To keep a page out of search results, allow crawling and add a noindex meta tag, or return an x-robots-tag header.
Where do I put the robots.txt file?
At the very root of your domain, so it lives at https://yoursite.com/robots.txt. It only applies to the exact host and protocol it sits on, so a separate one is needed for subdomains.
Should I add my sitemap to robots.txt?
Yes. Adding a Sitemap line helps every search engine discover your sitemap even if you have not submitted it in their console. You can list more than one.
What does crawl-delay do?
It asks crawlers to wait a number of seconds between requests. Google ignores it (control Googlebot in Search Console instead), but Bing and others honor it. Leave it blank unless a crawler is overloading your server.
Keep exploring SmashSERP
Crawlable is step one. Citable is the win.
SmashSERP audits your technical SEO and gets your pages cited by AI answer engines, on one plan.
One plan, $199/month. No contract, cancel anytime.