Robots.txt Generator

Generate a robots.txt file for your website. Common directives, custom rules, crawl delay, and sitemap URL support.

User-Agent

Crawl Delay (seconds)

Common Directives

Allow all pagesAllow: /Disallow /adminDisallow: /adminDisallow /privateDisallow: /privateDisallow /apiDisallow: /apiDisallow /tmpDisallow: /tmpDisallow /cgi-binDisallow: /cgi-binDisallow /*.pdf$Disallow: /*.pdf$Disallow /searchDisallow: /search

Custom Rules (one per line)

Sitemap URL

Generated robots.txt

User-agent: *

Frequently Asked Questions

What is robots.txt?▼

A file that tells search engine crawlers which pages they can or cannot access on your website.

Related Tools

Sitemap Generator

Generate XML sitemaps from a list of URLs. Set change frequency, priority, and last modified dates. Copy or download the output.

Meta Tag Generator

Generate meta tags for your website with live Google search preview. Create title, description, keywords, robots, and viewport meta tags.

Schema Markup Generator

Generate JSON-LD schema markup for Article, Product, FAQ, LocalBusiness, and Organization types. Improve your search results.