You open your server logs on a Monday morning and, alongside standard Googlebot traffic, you see requests from GPTBot, ClaudeBot, PerplexityBot, and CCBot. Your marketing team wants to appear in AI answers; your legal team is concerned about price tables, customer documents, and licensed reports being used for model training. The question is: How much access should you grant to each bot?
The short answer: The healthiest approach to AI bot access is not to block or allow all bots indiscriminately, but to score each bot based on purpose, citation potential, traffic load, content risk, revenue model impact, and measurability. GPTBot and ClaudeBot should be considered primarily in the context of model training or improvement; PerplexityBot may carry higher citation potential regarding direct answers and source discovery; while CCBot should be handled separately due to the risk of broad reuse via Common Crawl.
Why a single robots.txt decision is not enough
The most common mistake when writing an AI bot policy in robots.txt is treating all bots as if they have the same intent. However, a decision for GPTBot does not yield the same business outcome as a decision for CCBot. One bot might crawl a page for future model training, another for source discovery in an answer engine, and a third might include the data in a massive open web archive.
Furthermore, robots.txt is not a firewall. It is a public file that tells compliant crawlers which paths not to crawl. Confidential pricing, contracts, customer panels, health data, or membership content are not protected by robots.txt; these areas require session authentication, authorization checks, and, if necessary, IP-based access policies. The purpose of this matrix is to establish a publishing policy rather than security: making measured decisions on which public content to expose to which AI bot.
The practical roles of the four bots
GPTBot
GPTBot is the web crawler user agent used by OpenAI and is generally associated with model development processes. Allowing a page to GPTBot does not mean that page will be cited as a source in a specific ChatGPT answer. Therefore, when deciding for GPTBot, you should weigh whether you want your brand's open web information represented in the model ecosystem rather than expecting direct traffic gains.
ClaudeBot
ClaudeBot is the crawler user agent for Anthropic. The decision to allow ClaudeBot, similar to GPTBot, is a choice regarding whether public information, guides, documentation, and brand narratives should be learnable by AI systems. However, a stricter permission logic should be used for paid databases, comparative reports, customer case studies, and licensed content.
PerplexityBot
Because PerplexityBot is more closely associated with AI answer and search experiences, it may be seen as having higher citation and discovery potential for publishers. This is still not a guarantee of visibility. It would be unwise to grant broad permissions without isolating PerplexityBot traffic in your logs to monitor which URL types it visits, response codes, and crawl frequency.
CCBot
CCBot is the crawler for Common Crawl. Common Crawl data can be used for research, search, language model training, and various data processing pipelines. For this reason, while CCBot's direct short-term citation contribution may be low, the scope of content reuse is broad. A more protective default decision logic is advisable for sites publishing licensed reports, product data, listings, catalogs, or unique datasets.
6-criteria AI bot access matrix
Score the matrix below from 1 to 3 for each bot. 1 means low suitability or high risk, 2 means a medium situation requiring monitoring, and 3 means a positive signal for controlled access. If the total score is 14 or higher, consider selective access; if between 10-13, consider narrow access and log monitoring; if 9 or below, consider blocking or allowing only specific folders.
| Criterion | What is measured? | Low score signal |
|---|---|---|
| Purpose alignment | Does the bot's intent align with your publishing goals? | High weight of content you don't want in training data |
| Citation potential | The bot's relationship with answer engines or discovery | Low expectation of direct source traffic |
| Traffic load | Request count, repeated crawling, 4xx and 5xx rates | CPU spikes, cache misses, heavy PDF crawling |
| Content risk | Sensitivity regarding license, privacy, and competition | Paid reports, customer docs, private pricing |
| Revenue model impact | Dependency of content on leads, subscriptions, or sales | Bot scraping core revenue content for free |
| Measurability | Can user-agent, IP, response code, and URL paths be tracked? | No logging or inability to isolate bot traffic |
Initial decision table
| Bot | Recommended starting point | Reason |
|---|---|---|
| GPTBot | Selective access | Public guides and product docs can be opened; pricing, panels, and licensed files should be closed. |
| ClaudeBot | Selective access | Controlled access can be granted for brand and expertise content; private customer material should be excluded. |
| PerplexityBot | Allow and monitor | Higher potential for source discovery in AI search; monitor load and URL quality in logs. |
| CCBot | Narrow access or block | Risk increases for licensed and commercial datasets due to the broad reuse potential of Common Crawl data. |
Technical checklist
- Generate separate reports for GPTBot, ClaudeBot, PerplexityBot, and CCBot requests in your access logs from the last 30 days.
- Extract URL breakdowns for each bot: group blog, guide, product, search, cart, panel, PDF, and API paths separately.
- Measure 2xx, 3xx, 4xx, and 5xx response rates; if you see 5xx, resolve server load issues before making access decisions.
- List public and strategic content: evergreen guides, glossaries, documentation, and brand pages are usually candidates.
- Identify risky paths:
/musteri-paneli/,/teklifler/,/sepet/,/arama/,/api/, and private PDF folders. - Write separate robots.txt groups for each bot; do not rely on a single
User-agent: *block. - After changing robots.txt, verify the file returns a 200 status at
https://alanadiniz.com/robots.txt. - Verify behavior via logs for important rules; some crawlers may apply rules with a delay.
- Use authentication and authorization checks instead of robots.txt for pages requiring real protection.
- Review the policy quarterly; bot names, product intents, and your site's revenue model may change.
Example scenario: Bot-based policy on a B2B software site
Assume a B2B software company has three content types: public guides under /blog/, product usage documentation under /rehberler/, and paid research PDFs provided after a lead form under /pdf/ozel-raporlar/. There are also /musteri-paneli/, /teklifler/, and internal search results. For this site, opening the blog and guides to GPTBot and ClaudeBot, keeping PerplexityBot allowed while monitoring it more closely, and limiting CCBot only to the blog could be a reasonable starting point.
User-agent: GPTBot
Allow: /blog/
Allow: /rehberler/
Disallow: /musteri-paneli/
Disallow: /teklifler/
Disallow: /pdf/ozel-raporlar/
Disallow: /arama/
Disallow: /api/
User-agent: ClaudeBot
Allow: /blog/
Allow: /rehberler/
Disallow: /musteri-paneli/
Disallow: /teklifler/
Disallow: /pdf/ozel-raporlar/
Disallow: /arama/
Disallow: /api/
User-agent: PerplexityBot
Allow: /blog/
Allow: /rehberler/
Disallow: /musteri-paneli/
Disallow: /teklifler/
Disallow: /pdf/ozel-raporlar/
Disallow: /arama/
Disallow: /api/
User-agent: CCBot
Allow: /blog/
Disallow: /blog/ozel-raporlar/
Disallow: /
In this example, Disallow: / provides a general block for CCBot, while Allow: /blog/ leaves only the blog path open. The longest match logic is important in rule interpretation; /blog/ozel-raporlar/ remains closed because it is more specific. Still, this file does not protect sensitive data. The customer panel must be protected with session authentication, and PDFs should not be left public with predictable URLs.
Common mistakes
- Managing all AI bots under
User-agent: *and losing bot-specific nuances. - Interpreting GPTBot access permission as a direct result of appearing in AI answers.
- Treating CCBot like an ordinary search engine crawler and ignoring the scope of data reuse.
- Leaving internal search, filters, carts, and parameterized URLs open, wasting crawl budget.
- Mistaking robots.txt for secret content protection.
- Making decisions based on guesses about traffic load without log measurement.
- Writing the policy once and failing to update it when bot behavior or business models change.