When you check your server logs on a Monday morning, you notice CPU usage spiking to 90% and your website's response time exceeding 5 seconds. You might think you are under attack; however, what is actually happening is OpenAI's GPTBot or Anthropic's ClaudeBot starting a new training cycle by sending hundreds of requests per second to crawl your site. With traditional SEO knowledge, you immediately add 'Crawl-delay: 10' to your robots.txt file, but the crawl rate doesn't change. Why?
Short Answer: Is Crawl-delay a Solution for AI Bots?
No, the Crawl-delay command is not supported by the vast majority of modern AI bots (GPTBot, ClaudeBot, CCBot, etc.). Although this directive was standardized for search engines like Bing and Yahoo in the early 2000s, today's AI agents ignore this command. To control AI bot access speed, you need to use server-level (Nginx/Apache) rate limiting or WAF (Web Application Firewall) rules instead of robots.txt.
Why is Crawl-delay Insufficient in the AI Era?
Traditional search engine bots operated within a certain rule of 'politeness' to index the web. However, AI bots are distributed systems designed to collect massive datasets in the shortest time possible. The Crawl-delay command has no official requirement in a standard RFC document (robots.txt protocol); it is merely a suggestion left to the discretion of bot developers.
1. GPTBot (OpenAI) and Rate Limiting
GPTBot, operated by OpenAI, fully complies with Allow and Disallow rules in the robots.txt file. However, there is no mention of Crawl-delay support in its official documentation. While crawling your site, GPTBot may arrive at a speed (RPS - Request Per Second) that strains your server's capacity. Although OpenAI shares the bot's IP ranges, it does not allow you to set the speed via a user panel or robots.txt.
2. ClaudeBot (Anthropic) Behavior
ClaudeBot similarly does not recognize the Crawl-delay command. Although Anthropic claims to use its own internal algorithms to prevent the bot from overloading the server, this 'automatic speed adjustment' is often insufficient on shared servers. If you want to restrict ClaudeBot access, User-Agent based blocking or server-side limit_req configuration is essential.
3. CCBot (Common Crawl) and Data Pool
Common Crawl (CCBot), used as a source by more than 80% of AI models, provided limited support for the Crawl-delay command in the past, but in modern crawling architecture, the impact of this command is less than 5%. Since CCBot comes through thousands of distributed IPs, setting a single-second limit becomes technically impossible.
4. The OAI-SearchBot (SearchGPT) Difference
OpenAI's real-time search bot, OAI-SearchBot, is different from the training bot (GPTBot). This bot arrives to find answers to users' instant queries. Slowing down this bot may cause your site not to appear in SearchGPT results or to be presented with outdated information. Therefore, it is critical to distinguish the bot's purpose when setting rate limits.
Technical Solution: Server-Level Rate Limiting
If you really want to take control of AI bot access speed, you should take the reins instead of leaving it to the bot itself. Here are the most effective methods:
User-Agent Based Limiting with Nginx
On your Nginx server, you can allow a maximum of 1 or 2 requests per second for specific bots only. This prevents your site from crashing while allowing the bot to retrieve data, albeit slowly.
Cloudflare WAF Rules
If you are using Cloudflare, you can identify AI bots under the 'Bot Management' tab and set the maximum number of requests these bots can make within 10 seconds using 'Rate Limiting' rules. This stops the bot at the edge before it puts a load on the server.
Technical Checklist: AI Bot Management
- User-Agent Detection: Analyze your logs to determine which bot (GPTBot, ClaudeBot, CCBot, PerplexityBot) is consuming how much traffic.
- robots.txt Update: Use
Disallow: /if you want to block them completely; however, switch to server configuration if you want to slow them down. - Nginx limit_req_zone: Set up a dedicated memory zone for bots.
- HTTP 429 Response: Return the '429 Too Many Requests' error code to bots when they exceed the limit. Modern bots automatically reduce their crawl speed when they receive this code.
- IP Whitelisting: Keep SEO-critical bots like Googlebot and Bingbot exempt from these limitations.
Example Scenario: Nginx Rate Limit Configuration
The following code block sets a 1 request per second limit for GPTBot and ClaudeBot only, without affecting normal users. You can add this code to the http block of your nginx.conf file:
map $http_user_agent $ai_bot {
default "";
"~*GPTBot" "gpt";
"~*ClaudeBot" "claude";
}
limit_req_zone $ai_bot zone=ai_limit:10m rate=1r/s;
server {
location / {
limit_req zone=ai_limit burst=5 nodelay;
# Other configurations
}
}In this configuration, rate=1r/s indicates that the bot can crawl only 1 page per second, and burst=5 indicates that a tolerance of up to 5 requests is allowed for instantaneous spikes.
Common Mistakes
- Relying on Crawl-delay: Simply writing this command and expecting the server to relax is the biggest mistake.
- Using User-agent: * : Applying rate limits to all bots (including Google) can paralyze your SEO performance.
- IP Blocking: AI bots use thousands of different IPs; blocking a single IP is not a permanent solution.
- 403 Forbidden Error: Returning a 429 error instead of blocking the bot completely ensures the bot understands you are not 'malicious'.