A content team publishes a new guide at night, and by morning the AI answers are still showing the old page. The team immediately starts digging through robots.txt, but the real question is this: can GPTBot and ClaudeBot actually fetch that URL, or is the path being blocked by the server, CDN, or a miswritten rule?
Short answer: you can't understand AI bot access from a single source. You need to read robots.txt rules, server/CDN logs, and test requests made with a bot user agent together before you can make a sound decision. The most practical way to verify GPTBot access and ClaudeBot permission status is to do it in 10 minutes.
10-minute check flow
1) Read the robots.txt file
The first check is the explicit instruction you give bots. robots.txt is not an authentication system; it is a set of directives for well-behaved crawlers. So don't interpret access policy before you look at the file.
curl -s https://www.ornek.com/robots.txt | grep -nE 'GPTBot|ClaudeBot|User-agent|Disallow|Allow'What you're looking for is this: is there a specific block for GPTBot or ClaudeBot, or are the general User-agent: * rules taking effect? If the file doesn't exist or returns 404, that means there is no custom allow/block definition; still, you need to check separately that the server isn't blocking access at another layer.
2) Send a quick request with the bot user-agent
The second step is to see how the server responds to that user-agent. This test does not prove that the bot follows robots.txt; it only shows how your server handles that user-agent. So the 200, 403, or 404 you see here is a signal about access policy, not the final verdict on its own.
curl -s -o /dev/null -D - -A 'GPTBot' https://www.ornek.com/ozel-sayfa
curl -s -o /dev/null -D - -A 'ClaudeBot' https://www.ornek.com/ozel-sayfaRead the HTTP status codes like this: 200 means the content is being served; 301/302 means there is a redirect; 403 means the server or a WAF rule is blocking it; 404 means the page does not exist. If you allowed it in robots.txt but get 403 here, the problem is not robots.txt, but most likely the security layer.
3) Verify real requests in the logs
The most valuable data for AI bot access is server logs. Because this is where you can see what was actually requested and when. If you're using a CDN, check CDN logs; if you're looking directly at the server, check access logs.
grep -Ei 'GPTBot|ClaudeBot' access.log | tail -n 100If you see the relevant user-agent in the logs, it means the bot tried to reach that page at least once. If there is no record at all, that does not mean it definitely could not access it; sometimes crawl frequency is low, and sometimes traffic is sampled at the CDN layer. That's why you should inspect not just a few hours of data, but, if possible, the last few days.
4) Separate robots.txt from a server block
A very common mistake is to assume the whole problem is robots.txt. In reality, three layers can do different things: robots.txt gives crawl instructions, the server returns a response, and the WAF/CDN cuts off the request. A page may be open in robots.txt but still return 403 because of the WAF; the reverse is also possible.
| Check | What it means | What you should do |
|---|---|---|
| Disallow in robots.txt | The bot is being told not to crawl that path | Fix the rule if you want to allow it |
| 200 with curl | The server is serving that request | This does not prove the bot read robots.txt |
| 403 with curl | The server or security layer is blocking it | Review the WAF, CDN, IP, or rule set |
| No request in logs | The bot may never have reached that record | Confirm with a different date range and CDN logs |
Why this check matters
AI systems can use different crawlers, different frequencies, and different URL priorities. One page may appear in search results while another does not; the reason is often not content quality, but a difference in the access layer. A few lines changed in robots.txt can significantly alter crawl behavior, especially on news, product, and campaign pages.
The goal here is not to blindly block or open bots; it is to measure first which bot can actually reach which URL. If you make changes without measuring, you're more likely to create new problems instead of solving existing ones. So first check, then policy.
Technical checklist
- Check the specific rules for GPTBot and ClaudeBot in robots.txt.
- Make sure the general User-agent: * rules do not override the specific ones.
- Send the GPTBot and ClaudeBot user-agent with
curlto the relevant URL. - Look for these user-agents in server, CDN, and WAF logs.
- If 403 conflicts with robots.txt, isolate which layer is blocking it.
- Test not only one URL, but also a sample folder: homepage, category, detail page.
- Repeat the test at different times; crawl frequency is not fixed.
Example scenario: access testing on a news site
Let's say archive pages on a news site are not showing up in AI answers. First, consider this robots.txt setup:
User-agent: GPTBot
Disallow: /arsiv/
User-agent: ClaudeBot
Disallow: /premium/
Allow: /With this setup, GPTBot should not enter the archive folder, while ClaudeBot should be able to see other pages except premium. Then you run these tests against the same URLs:
curl -s -o /dev/null -D - -A 'GPTBot' https://www.site.com/arsiv/2024/01/manset
curl -s -o /dev/null -D - -A 'ClaudeBot' https://www.site.com/premium/analiz
curl -s -o /dev/null -D - -A 'ClaudeBot' https://www.site.com/haber/ekonomiWhat you should look at here is whether the server response is consistent with the robots.txt instructions. If you see 403 on a page that should be open to ClaudeBot, the problem is most likely not robots.txt; it's probably the WAF, a CDN rule, or IP-based filtering. If GPTBot never shows up in the logs for the archive folder, you need to reconsider not just the crawl directive, but also the crawl request or discovery links.
Common mistakes
- Changing robots.txt and then not checking the logs.
- Interpreting a 200 response from curl as proof that the bot definitely saw the page.
- Searching for GPTBot or ClaudeBot in Search Console; that tool is for Googlebot.
- Looking at only one user-agent and missing other AI crawlers.
- Confusing a CDN/WAF block with robots.txt.
In short: AI bot access is not something you solve in a single line, but with the right three-part check, you can clarify it in 10 minutes. First robots.txt, then the test request, and finally log verification. If you keep that order, you can reliably see which bot can really reach which URL.