Short answer
AI company bots do three different jobs: training models, building search indexes, and fetching pages for a user in the moment. Every major lab now documents separate bots per purpose, so you can block training while keeping visibility in AI answers. User-triggered fetchers generally ignore robots.txt, though, and Cloudflare introduces new default blocks on September 15, 2026.
Whether to block AI bots has become three questions, because the bots now do three different jobs with three different consequences. Treating them as one decision usually means blocking the wrong thing.
Which bots come, and what do they do?
OpenAI documents four agents: GPTBot collects training data, OAI-SearchBot builds the search index, ChatGPT-User fetches pages at a user’s request, and OAI-AdsBot reviews ads. The documentation is blunt about the consequence: a site that shuts out OAI-SearchBot will never appear in ChatGPT search answers, while a blocked GPTBot only stops training. Anthropic makes the same split with ClaudeBot, Claude-SearchBot and Claude-User, and Google separates Googlebot, the training control Google-Extended, and the new agent bot Google-Agent from March 2026.
The split makes the decision manageable: blocking training and keeping visibility are two different lines in robots.txt. Meta documents the same three-way division in its crawler documentation, which also documents the search index bot Meta-WebIndexer.
Do the bots obey robots.txt?
Partly, and the difference is documented. Google’s crawler documentation states that user-triggered fetches generally ignore robots.txt because a person asked for the page, and OpenAI and Perplexity write the same about their counterparts. TollBit’s measurement of Q4 2025, reported by Digiday in February 2026, additionally found 30% of AI scrapes going past robots.txt rules, and a full 42% for ChatGPT-User. Robots.txt is a statement of intent; real blocking takes a firewall or a CDN.
What is changing in the infrastructure?
- Cloudflare has sorted AI traffic into search, agent and training categories since July 2026, and blocks training and agents by default for new ad-carrying domains from September 15, 2026
- The IETF standard for AI preferences, with authors from Google and Mozilla, reaches its final milestone in August 2026 with a Content-Usage rule for robots.txt
- Web Bot Auth lets bots identify themselves cryptographically, and Google has been testing the protocol for its agents since May 2026
- Payment rails like Cloudflare’s pay-per-crawl remain in closed beta, and no AI giant yet pays openly through the RSL licensing standard
The much-discussed llms.txt file belongs in the same sober column: Ahrefs examined 137,000 domains in June 2026 and found 97% of the files never receive a single request, and Google has repeated that Search ignores it.
Is your website built for agents?
Agents running errands for users became everyday reality in 2026: Perplexity’s Comet is free and on iPhone, Claude clicks and fills in forms through the browser on every paid plan, and Google’s Gemini handles web tasks with the user’s logins. Google’s own guidance for agent-friendly sites is concrete: real buttons and links in semantic HTML, stable layouts, and forms with proper labels, because agents read the DOM, screenshots and the accessibility tree. All of it makes the site better for people too.
Common questions
- Which AI bots should I allow if I want customers from AI answers?
- The search index and user-fetch bots: OAI-SearchBot and ChatGPT-User for ChatGPT, Claude-SearchBot and Claude-User for Claude, PerplexityBot for Perplexity, plus Googlebot and bingbot. Training bots like GPTBot and ClaudeBot are a separate decision with no link to visibility.
- Does robots.txt actually stop AI bots?
- Only the declared crawlers. User-triggered fetches generally ignore the file per Google’s, OpenAI’s and Perplexity’s own documentation, and TollBit measured 30% of AI scrapes going past the rules in Q4 2025. To truly shut traffic out, you need firewall or CDN rules.
- Can Cloudflare start blocking AI bots for me automatically?
- Yes. For new domains and new ad-carrying sites, training and agent bots are blocked by default from September 15, 2026, while search bots pass through. If your site sits behind Cloudflare, review the settings so the default is a choice you actually made.
- Can I charge AI companies for reading my content?
- The rails exist but the money is small. Cloudflare’s pay-per-crawl is in closed beta, the RSL licensing standard has infrastructure firms and publishers behind it but no openly paying AI giant yet, and Perplexity’s revenue share covers enrolled publishers. For most companies, visibility is still the entire compensation.
- How do I know a bot is who it claims to be?
- Check the sender against the companies’ published IP lists, like OpenAI’s and Anthropic’s JSON files. The next step is cryptographic identification through Web Bot Auth, which Google has tested for its agents since May 2026 and Cloudflare already uses for signed agents.
- Can my website handle AI agents that click and fill in forms?
- Test it yourself: can the site’s main flow be completed by keyboard, do buttons and links use real elements and labels, and does content hold still while the page loads? Google’s agent-friendly guidance overlaps almost entirely with accessibility work, so the effort pays twice.
Sources
- OpenAI bots and crawlers — OpenAI developer docs
The documentation of GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot, and the rule that blocking OAI-SearchBot removes a site from ChatGPT search answers.
- Does Anthropic crawl data from the web? — Anthropic Help Center
Anthropic’s three-way split into ClaudeBot, Claude-SearchBot and Claude-User, with robots.txt control and IP lists.
- Google user-triggered fetchers — Google Crawling Infrastructure
Google-Agent and the statement that user-triggered fetches generally ignore robots.txt, updated 2026-08-19.
- Perplexity crawlers — Perplexity docs
PerplexityBot for the search index, and Perplexity-User generally ignoring robots.txt on user fetches.
- Your site, your rules: new AI traffic options — Cloudflare
The search, agent and training categories, and the default blocks arriving September 15, 2026.
- AI licensing deals, protection measures aren’t slowing web scraping — Digiday
TollBit’s measurement: 30% of AI scrapes bypassed robots.txt in Q4 2025, 42% for ChatGPT-User.
- We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read — Ahrefs
The June 2026 study of 137,210 domains where 97% of llms.txt files never received a request.
- Build agent-friendly websites — web.dev
Google’s agent-friendly guidance: semantic HTML, stable layouts, and agents reading the DOM, screenshots and the accessibility tree.
Tell us what you want to build
Thirty minutes, free of charge, and a straight answer on whether we are the right studio for it.