Robots.txt Generator
Generate a robots.txt file for your website. Common directives, custom rules, crawl delay, and sitemap URL support.
User-agent: *
What is Robots.txt Generator?
A robots.txt generator creates the file that controls which pages search engine crawlers can and cannot access on your website. Every website should have a robots.txt at its root — it's the first file search bots look for. Proper configuration prevents indexing of admin pages, staging environments, duplicate content, and internal search results while ensuring your important pages are fully crawlable. Mistakes in robots.txt can accidentally block your entire site from Google.
How It Works
Select which user-agents to configure (Googlebot, Bingbot, all bots), then specify allow and disallow rules for URL paths. The tool generates valid robots.txt syntax following the Robots Exclusion Protocol standard. You can add sitemap references, set crawl-delay directives, and create separate rules for different crawlers. The output shows the complete robots.txt content ready to upload to your site root, with comments explaining each directive.
Common Use Cases
- •Blocking search engines from indexing admin panels, login pages, and internal tools
- •Preventing duplicate content indexing from URL parameters, pagination, and filtered views
- •Configuring different access rules for different search engine bots (Google vs Bing vs others)
- •Adding sitemap location references to help search engines discover your content structure
- •Blocking staging and development environments from appearing in search results
- •Preventing crawling of large media files or API endpoints that waste crawl budget
Frequently Asked Questions
What is robots.txt?▼
A file that tells search engine crawlers which pages they can or cannot access on your website.
Does robots.txt stop a page appearing in Google?▼
No, and this trips people up constantly. It asks crawlers not to fetch the page; the URL can still be indexed if other pages link to it, appearing without a description. To keep a page out of results use a `noindex` meta tag and leave the page crawlable so the tag can be read.
Is robots.txt a security control?▼
Absolutely not. It is a public file that anyone can read, and listing your admin paths in it is effectively publishing a map of what you consider sensitive. Protect private areas with authentication. Well-behaved crawlers obey it; malicious ones read it for ideas.
Where must the file be located?▼
At the root of the host, `https://example.com/robots.txt`. It applies only to that exact protocol, subdomain and port, so a file at the apex domain does not govern `blog.example.com`, which needs its own.
How do Allow and Disallow interact?▼
Google applies the most specific matching rule rather than the first, so a longer path wins. That lets you block a directory but permit one file inside it — `Disallow: /assets/` together with `Allow: /assets/public.css` works as intended. An empty `Disallow:` value permits everything.
Should I block CSS and JavaScript files?▼
No. Google renders pages to assess them, so blocking your stylesheets and scripts means it sees a broken layout and may judge the page unusable on mobile. This was a common recommendation years ago and is now actively harmful.
Does crawl-delay work with Google?▼
Google ignores `Crawl-delay` and expects crawl rate to be managed in Search Console instead. Bing and Yandex do honour it. Including the directive is harmless but has no effect on Googlebot.
Related Tools
More from SEO Tools:
Sitemap Generator
Generate XML sitemaps from a list of URLs. Set change frequency, priority, and last modified dates. Copy or download the output.
Meta Tag Generator
Generate meta tags for your website with live Google search preview. Create title, description, keywords, robots, and viewport meta tags.
Schema Markup Generator
Generate JSON-LD schema markup for Article, Product, FAQ, LocalBusiness, and Organization types. Improve your search results.