Free tool · Website & technical SEO

Robots.txt Generator

Generate a safe robots.txt with presets.

Start each path with “/”. Don’t block CSS, JS or images — Google needs them to render pages.

AI crawlers

Blocking AI search bots (e.g. OAI-SearchBot, PerplexityBot) can stop your site being cited in AI answers.

Google ignores crawl-delay; Bing and some others respect it.

robots.txt

    How to use it

    1. Pick a preset for your platform.
    2. Add paths to block and your sitemap URL.
    3. Choose whether to allow AI crawlers.
    4. Download and upload to your site root, then test in Search Console.

    Good to know: robots.txt controls crawling, not indexing. To keep a page out of Google, use a noindex tag instead.

    Everything runs in your browser. Nothing you enter is sent to 404 Marketers.