robots.txt AI Bot Test: Which AI Bots Are You Allowing?
Your robots.txt may be blocking GPTBot without you knowing — or a bot you allow may be stopped at the CDN layer. Index Harmony resolves your file bot by bot, then sends real requests with AI bot identities and reports whether what is written matches what actually happens.
No sign-up · 30–90 seconds · 100+ checks
What robots.txt says vs what bots actually get
Most tools only read your robots.txt. We measure both what the file says and what bots really experience — because the two often differ:
Line-by-line analysis
We fetch your robots.txt and resolve the effective rule for every AI bot — including group rules, Allow/Disallow precedence and wildcards.
Real bot requests
We do not stop at parsing: from GPTBot to ClaudeBot, we send real HTTP requests to your site with real AI bot identities and record each response.
CDN/WAF behavior
Even when robots.txt allows a bot, a CDN or firewall can turn it away at the door. Every gap between the written permission and the real response becomes a finding.
What do we check?
Per-bot permission status
The decision your robots.txt reaches for each AI bot — GPTBot, ClaudeBot, PerplexityBot and more: allowed or blocked?
Indirect wildcard blocks
A generic Disallow under "User-agent: *" may be silently shutting out every AI bot that has no rule of its own.
Sitemap declaration
Does your robots.txt declare a Sitemap line, and is the declared URL actually reachable?
Content-Signal declaration
Is the new Content-Signal present — the signal that lets you state search, AI-answer and model-training permissions separately?
Soft-404 detection
Does your robots.txt URL return 200 while actually serving an error page? One of the most misleading situations for bots.
Who is it for?
Sites behind a CDN/WAF
If you use a layer like Cloudflare, the permission in the file alone is not enough — see whether the bot can actually get in.
Publishers
Decide which bot may use your content and for what purpose, with a deliberate robots.txt and a Content-Signal declaration.
E-commerce sites
Make sure a generic Disallow is not accidentally cutting AI shopping assistants off from your product pages.
Agencies and SEO specialists
Document your clients' AI bot permissions in a single report; re-test after every change.
Frequently asked questions
What is a robots.txt AI bot test?
A test that measures what your robots.txt tells AI crawlers like GPTBot, ClaudeBot and PerplexityBot — and what those bots actually receive. Index Harmony resolves the file bot by bot, then sends requests to your site with real bot identities and compares the two outcomes.
How do I allow GPTBot in robots.txt?
Add "Allow: /" under a "User-agent: GPTBot" line; to block it, use "Disallow: /". Careful: when GPTBot has no rule of its own, a generic Disallow under "User-agent: *" applies to it as well. The scan shows exactly which decision your file reaches for GPTBot.
How do I know if I am allowing ClaudeBot?
Run a free scan: the report shows your robots.txt decision for ClaudeBot and its search/user variants (Claude-SearchBot, Claude-User) separately from whether the bot can actually reach your site.
Can bots still be blocked even though robots.txt allows them?
Yes — one of the most common findings. CDN, WAF or server rules can cut off bot requests regardless of robots.txt. That is why we do not stop at reading the file: we send real requests with each bot identity and report the response it receives.
Should I block AI bots or allow them?
It is a trade-off. Allowing them means visibility and a chance to be cited by AI assistants; blocking them gives you more control over how your content is used. You do not have to choose just one: with a Content-Signal declaration you can allow search visibility while separately opting out of model training. The report shows where you stand today — the decision is yours.
See what your robots.txt really does
Enter your URL — your per-bot permission table and 100+ check AI-readiness report are ready within 90 seconds.