One wrong line in robots.txt can deindex a site — 'Disallow: /' has ended careers. This generator builds a correct file from safe presets (allow-all, staging block, WordPress) plus your custom paths, a Sitemap line, and optional explicit allows for AI crawlers like GPTBot and ClaudeBot.
User-agent: * Disallow:
In the AI-search era, robots.txt is also your answer-engine policy: allowing AI crawlers makes your content quotable in ChatGPT, Claude, and Perplexity answers — free brand distribution for most publishers.
Compose robots directives without memorizing syntax.
URL list in, standards-valid sitemap.xml out.
Old→new pairs become ready 301 rules.
Built by GuestPostOn — the free guest post marketplace with 1,000+ verified sites, full SEO metrics, and transparent pricing. Browse verified guest post sites →