How to design responses for AI bots with 200, 403, and 429: a 12-URL test plan

AI bot HTTP status codes are HTTP responses that tell an artificial intelligence crawler whether the web server is serving content and when it should try again. On a product site, a crawler like GPTBot may be able to read some pages while a WAF layer may stop another request with 403. The short answer is this: use 200 for accessible and readable content, 403 for requests denied by policy, and 429 for temporary rate limiting; verify each outcome with 12 separate test URLs, response headers, and server logs.

This approach is separate from the decision to allow or block the AI bot itself. If you do not distinguish between permanent access denial and temporary congestion control for the same bot, the crawler may retry unnecessarily, never receive the content, or mistake a 200 response from a blocking page for the real page.

What does a 200 response mean for an AI bot?

200 means the server successfully processed the request and returned a response body. This is the right starting point for a crawler to retrieve content; however, it does not guarantee indexing, citation in an AI answer, or repeated crawling.

A page returning 200 should at least contain the correct content, an appropriate Content-Type, and a consistent body. For an HTML page, text/html is typically used. A login screen, WAF warning, or empty template should not be delivered with 200. The crawler will see a successful HTTP-layer response, but if the content is not the expected document, downstream interpretation can break.

How do you verify a 200 test?

Measure the status code, redirect chain, Content-Type value, response size, and first content bytes with a GET request. Repeat the test at least 3 times; also check whether the CDN cache and origin server return different responses.

When should 403 be used for bot blocking?

403 means the server understood the request but is refusing access. It is appropriate when you are returning a permanent or conditional denial due to authentication policy, a WAF rule, or a specific bot access policy; 429 should be preferred over 403 when you want to express temporary traffic pressure.

A crawler receiving 403 cannot reliably read the page body. Some crawlers may reduce retries, while others may try again after a certain period; this behavior varies by bot provider. For that reason, do not assume crawling will definitely stop when you see 403. Review the same User-Agent, IP range, time, and rule match in server logs.

robots.txt and HTTP 403 are not the same thing. robots.txt is a crawling directive; 403 is an actual access denial from the server. If the bot cannot read robots.txt or already knows the page from another source, permission in robots.txt alone does not mean the content is reachable. Adding a noindex meta tag to a 403 page may also be ineffective if the crawler cannot get the content in the first place.

How should 429 rate limiting be designed for AI bot access?

429 indicates that the client sent too many requests within a given time window. This response is for cases where access is possible in principle but the current request rate is not accepted. Adding a Retry-After header to the response is the standard way to specify the waiting period; do not assume the crawler will strictly follow it.

The Retry-After value can be given as a number in seconds or as an HTTP date. For example, Retry-After: 60 tells the client to wait 60 seconds according to your policy. If a CDN mistakenly caches a 429 response, later low-rate requests may also receive the same error. For that reason, define an appropriate Cache-Control policy and CDN rule for temporary rate-limit responses.

Keep the 429 body short and machine-readable. If the status code is 429 but the body contains an HTML error page that returns 200 internally, the client may think it received a successful response. Binding the rate-limit key only to User-Agent is also weak; IP, verified bot identity, account, endpoint, and time window can be evaluated together.

What behavior should be tested for AI bot access?

The test should measure not only the status value, but also how the same request responds at the edge, at the origin, and at different times. Changing the User-Agent header does not prove real bot verification; it only simulates how your rule reacts to that header. If the provider publishes IP verification methods, use those as well.

How should a 12-URL test plan be set up?

To avoid risking production content, first create 12 URLs in staging or under a disposable /ai-test/ path. Document the expected status code, trigger condition, and header to watch for each URL in advance. The distribution below includes 4 cases for 200, 3 for 403, 3 for 429, and 2 recovery/control scenarios.

Which measurements should be used to verify the responses?

For each request, record the UTC time, HTTP status, number of redirects, response time, body size, Content-Type, Cache-Control, Retry-After, and Request-ID values. In the 429 test, report the rows separately before and after you actually apply the 60-second wait. In the 403 test, check whether the WAF event log and the application log show the same request.

curl -sS -D headers.txt -o body.html -A GPTBot https://example.com/ai-test/ok-01
curl -sS -D - -o /dev/null -A GPTBot https://example.com/ai-test/rate-03
curl -sS -L -w status=%{http_code}--time=%{time_total}n -o /dev/null -A GPTBot https://example.com/ai-test/forbidden-01

The first command saves the headers and body, the second displays the 429 headers directly, and the third follows redirects to measure the total result. -A GPTBot is only a User-Agent simulation; it does not prove you are a real bot. Before applying the rule in production, run the same commands from an allowed test IP.

Practical technical checklist

1. Verify that the 4 URLs expected to return 200 contain real HTML, the correct Content-Type, and a non-empty body.
2. Identify whether the denial decision for the 3 URLs expected to return 403 is made at the WAF, CDN, or application layer.
3. Document the rate-limit window, threshold, and key for the 3 URLs expected to return 429.
4. Add Retry-After in seconds or HTTP date format to the 429 response; confirm that this header is not altered by the CDN.
5. Do not use 403 for temporary spikes in traffic; keep permanent policy and temporary capacity issues separate.
6. Do not leave 200 responses on error, login, or CAPTCHA pages as if they were content pages.
7. Because GET and HEAD results may differ, measure the crawler scenario with GET; treat HEAD as a separate compatibility test.
8. Repeat the same test group at 3 different times, for example at 0, 60, and 180 seconds.
9. Store the results in a single CSV file with status code, response time, body size, and log request-id fields.

Example scenario: CDN and application both apply rate limiting

A news site wants to serve normal article HTML to allowed AI bot requests, close management paths with 403, and slow down a client sending excessive requests per second with 429. If the application and CDN do not use the same threshold, the CDN may generate 429 while the request never appears in the origin log. For that reason, observe URLs 8, 9, and 10 at both layers.

HTTP/1.1 429 Too Many Requests
Content-Type: text/plain; charset=utf-8
Retry-After: 60
Cache-Control: no-store

Rate limit reached. Retry after the indicated interval.

In this example, the value 60 is the wait time chosen by the site policy; it is not a universal value for all crawlers. In a 403 example, a similar header could be used, but adding Retry-After does not turn it into 429. At the end of the test, check that Retry-After is preserved on URL 10, that URL 11 returns 200 after the wait, and that the body is truly the article content.

Common mistakes

• Assuming every bot can be reliably verified with a single User-Agent string.
• Returning 403 for temporary traffic spikes and leaving it unclear when the crawler will retry.
• Returning 429 without sending Retry-After at all, or removing it in the wrong CDN layer.
• Replacing 403 and 429 responses with an access-block page that has a 200 status code.
• Testing only in the browser and not measuring real differences in User-Agent, redirects, and caching.
• Interpreting robots.txt permission as meaning the server will return 200.
• Drawing conclusions from a single request without 3 repeats and log comparison.

How should test results be interpreted?

200 results show that content delivery is working, 403 results show that the policy layer is active, and 429 results show that rate control has been triggered. None of them alone means AI search visibility. If the same URL returns different codes at different layers, first follow the CDN, WAF, and origin sequence; then evaluate the documented behavior of the actual bot provider separately.