Block AI Crawlers (robots.txt)
Pick which AI crawlers to block or allow, then copy the robots.txt rules. Everything runs in your browser.
robots.txt file at the root of your site (for WordPress, that is your site root, not wp-admin). These rules are advisory: well-behaved crawlers honor them, but they cannot force a crawler to comply. Rules are built in your browser and nothing is sent anywhere.About the Block AI Crawlers (robots.txt)
Block AI Crawlers generates robots.txt rules that tell AI and LLM crawlers, such as OpenAI GPTBot, Anthropic ClaudeBot, Google-Extended, and Common Crawl CCBot, whether they may access your site. Pick the crawlers you care about, choose to block or allow them, and copy the result into your robots.txt. It runs entirely in your browser.
How it works
- Choose a rule: block the selected crawlers, or allow only the selected crawlers and block the rest.
- Tick the AI crawlers you want the rule to apply to.
- Copy the generated robots.txt, or download the file.
- Place the rules in the robots.txt file at the root of your domain.
Features
- Covers the common AI crawlers from OpenAI, Anthropic, Google, Apple, Perplexity, and more.
- Two modes: block selected, or allow only selected.
- Select all or none with one click.
- Copy and download the robots.txt output.
- Runs client-side; nothing is uploaded.
Frequently asked questions
Does robots.txt actually stop AI crawlers?
It is a request, not a hard block. Well-behaved crawlers from major vendors honor robots.txt, but it cannot force compliance. For strict control, also block at the server or firewall level.
Where do I put these rules in WordPress?
They go in the robots.txt served at your domain root, for example https://example.com/robots.txt. Many SEO plugins let you edit robots.txt, or you can add a physical file in the site root.
What is the difference between GPTBot and Google-Extended?
GPTBot is OpenAI's crawler for training and indexing. Google-Extended controls whether your content is used for Google's Gemini AI, and it is separate from normal Google Search crawling, so blocking it does not affect your search ranking.
Will blocking these crawlers hurt my SEO?
No. These are AI-specific user-agents. Standard search crawlers like Googlebot and Bingbot are not in this list, so your normal search visibility is unaffected.
Is my data sent anywhere?
No. The rules are built in your browser. Nothing you select or generate is sent to a server.