robots.txt Generator: Control Crawlers, Switch AI Crawlers One by One

This free robots.txt generator builds the file that tells search engines and AI crawlers what they may fetch from your website. You block areas, add your sitemap and switch 21 AI crawlers one by one, each sourced from its provider. Then you check for any address whether a given crawler may read it. Below, I explain why I do not block good bots and where Cloudflare overrides your file.

Go to the generator

robots.txt generator with AI crawler switches

What is a robots.txt file and why do you need one?

A robots.txt file is a plain text file in the root directory of your website, for example at your-domain.com/robots.txt. Crawlers from search engines and AI providers read it before they fetch pages. It states which areas a crawler may visit and where your sitemap lives. The file only applies to its own host: a subdomain needs its own.

Without a robots.txt, crawlers may fetch everything. So the file is not mandatory. It becomes useful once you want to keep areas such as internal search or the cart out of crawling, or steer AI crawlers deliberately. A robots.txt is a request, not an access barrier. Reputable crawlers such as Googlebot follow it, others do not.

Create a robots.txt in four steps

With the generator above, your robots.txt file comes together like this:

1. Enter the areas no crawler should fetch and the address of your sitemap. The templates add common paths for WordPress and WooCommerce.
2. Decide for each AI crawler whether it may read your content. The quick choice sets all switches of a group at once.
3. Check a few important pages with the address check. The tool shows which group and which rule decides.
4. Save the file as UTF-8 in the root directory and review it in the robots.txt report in Google Search Console.

If you already have a robots.txt, paste it under "Import an existing robots.txt". Groups for crawlers the tool does not know stay unchanged.

Training, search, fetch: which AI crawlers exist

AI providers send different crawlers for different purposes. That is why the generator sorts them into three groups.

Training: GPTBot from OpenAI, ClaudeBot from Anthropic or CCBot from Common Crawl collect content for future models. Google-Extended and Applebot-Extended are not crawlers of their own but control tokens. If you block them, you stay in Google Search; according to Google, it has no effect on ranking.

AI search: OAI-SearchBot, Claude-SearchBot and PerplexityBot build the index that AI answers draw their sources from. If you block OAI-SearchBot, OpenAI says your pages no longer appear in ChatGPT's search answers.

Fetch on request: ChatGPT-User or Perplexity-User retrieve a page when someone asks for it in a chat. Perplexity states that this fetcher generally ignores robots.txt.

Why I do not block good bots

Training and visibility are two separate switches. Many block training to protect their content and leave AI search open. I often consider that the costlier mistake: a model that does not know your brand from training will not look for it specifically when someone searches. That is why litzki-systems.com explicitly allows training, with ai-train=yes and the major AI crawlers named in the robots.txt.

Stephen Burns ↗, Web Intelligence Lead at the Common Crawl Foundation, shows what happens without a deliberate decision, using Children's Hospital Los Angeles as an example ↗. A default setting at the hospital's provider blocked AI crawlers and CCBot. Parents searching in ChatGPT or Gemini for a clinic did not find the hospital.

The trend points the same way. The study Consent in Crisis ↗ counted for 2023 to 2024: within one year, about 5 percent of the tokens in the C4 training corpus became fully restricted, and at least 28 percent of the actively maintained sources. That also hits research, libraries and archives. If you only want to reduce server load, throttle instead of blocking: CCBot and Anthropic's crawlers follow Crawl-delay.

When Cloudflare decides differently from your robots.txt

Your robots.txt can allow everything and a crawler still does not get through. Cloudflare blocks AI crawlers at the network edge, regardless of the file. Since Sep 15, 2026, Cloudflare blocks training and agents by default on pages with ads for new domains. With "Bot Preference Sync", Cloudflare also puts its own lines in front of your robots.txt.

I measured this on Aug 27, 2026 across the 1,000 most visited websites on the Tranco list, with a named verification bot. Of the domains behind Cloudflare, 22.8 percent ↗ rejected it on first contact (28 of 123), compared with 4.7 percent for all other providers (27 of 572). If your website runs behind Cloudflare, tick the box in the generator and compare dashboard and file.

Check your robots.txt: the five most common mistakes

1. A group of its own overrides the rules for all. Each crawler follows only the group that names it most specifically. If GPTBot has its own group with "Allow: /", your blocks under "User-agent: *" no longer apply to it.
2. Disallow is not noindex. Google can still index a blocked page if other pages link to it. To keep a page out of the index, use the meta tag noindex, and for that the page must stay crawlable.
3. Case sensitivity: /Internal/ does not block /internal/.
4. Blocked CSS and JavaScript files: Google then renders and understands the page less well.
5. A "Disallow: /" from the staging site goes live and blocks the whole website.

The address check in the generator applies the same rules as Google and shows the deciding line. That way you find mistakes before search engines do.

Frequently asked questions

Is the robots.txt generator free?

Yes, without sign-up. The tool runs entirely in your browser, sends nothing to a server and sets no cookies. You paste an existing robots.txt by copying it, because the tool does not fetch other sites.

How do I change the robots.txt in WordPress or Shopify?

WordPress generates a robots.txt itself as long as there is no file in the root directory. Upload the file from the generator via FTP or your host's file manager, or use the file editor of your SEO plugin. In Shopify, you change it through the theme template robots.txt.liquid, which replaces the generated version entirely.

How long until changes take effect?

Google usually caches the robots.txt for up to 24 hours. OpenAI states about 24 hours for search, DuckDuckGo 72 hours for DuckAssistBot. In Google Search Console, the robots.txt report lets you request a recrawl.

Do all AI crawlers follow robots.txt?

No. The major providers commit to it for their training and search crawlers. Fetches on a person's behalf may not follow the file, according to OpenAI, Perplexity, Meta and Amazon. To really prevent access, you need rules on the server or in the firewall.

robots.txt, Cloudflare and AI visibility in one pass

I review robots.txt, firewall and server logs of your website together and show which crawlers actually get through. This is part of technical SEO and GEO. The initial call takes thirty minutes and is free and non-binding. You receive all findings in writing afterwards.

Book an initial call ↗ Prefer to write