Free robots.txt Generator for AI Bots

• Allow all AI bots gives every listed crawler the same access as your default group. Use it if you want the widest chance of being read, cited and trained on.
• Allow AI search, block training keeps OAI-SearchBot, Claude-SearchBot, PerplexityBot and the user-triggered fetchers open, and blocks the training crawlers (GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent) plus the Google-Extended and Applebot-Extended usage tokens.
• Block all AI bots disallows every listed agent. Your regular search visibility is unchanged because Googlebot and Bingbot still follow the * group.

"No rule" leaves a bot out of the file, so it follows User-agent: *. That's fine, but an explicit group makes your intent clear and protects you if someone later tightens the wildcard rules.

Crawlers pick the most specific group that names them, so the generator writes one wildcard group for everyone, one group listing all allowed AI agents and one listing all blocked agents. Several User-agent lines in a row share the rules below them, which keeps the file short. Private paths are repeated in the allowed group because a named group replaces the wildcard rules instead of adding to them — a detail that trips up many hand-written files.

After you publish, check the live file with the AI crawler checker. It reads robots.txt the way the bots do and shows the verdict for each one.

Blocking a training crawler keeps your future content out of that company's training data, but it doesn't remove what was already collected, and it doesn't stop other sites from quoting you. Blocking a search crawler is more costly: that engine can no longer fetch your pages to cite them, so answers about your category will cite someone else. If being recommended by AI matters to you, keep the search bots open even if you block training.