AI Bot Detection Behind a CDN: Logging the Real IP in 5 Steps

An e-commerce site's access logs showed 18,000 requests in a single night; all of them appeared to originate from Cloudflare IPs. The user-agent field contained fragments of GPTBot, ClaudeBot, and CCBot, but it was impossible to measure how many unique client IPs each bot was using. The problem wasn't with the bots, but with the origin server failing to log the real client IP behind the CDN.

Short answer: Before looking at the user-agent for AI bot detection, you must fix the client IP in your logs. If you use a CDN, the origin server often sees the CDN's proxy IP instead of the visitor's IP; the real IP is usually carried in the CF-Connecting-IP or X-Forwarded-For header. If you accept these headers only when they come from trusted CDN IP ranges and write them to your access log format, you can segment AI bot traffic in your server logs by IP, user-agent, URL, and status code.

Why fix the IP first?

AI crawler security decisions are often based on three signals read together: IP, user-agent, and request behavior. If you log all requests as if they came from a single CDN IP, you cannot see whether the same bot is using 3 IPs or 300. This leads to misinterpreting rate limit thresholds, the impact of robots.txt, and content crawling intensity.

For example, if there are 12,000 GPTBot requests in a daily log, the first question should be: How many unique real IPs did these requests come from? The second question: Is the same IP visiting specific URL patterns, or is it crawling the entire sitemap? The third question: Is the 403, 429, or 5xx rate concentrated on specific IPs? To answer these questions, you must store the CDN proxy IP and the real client IP in separate fields in your origin logs.

How do CDN and proxy headers work?

A reverse proxy or CDN receives the request from the visitor and opens a new connection to the origin server. Therefore, the source IP of the TCP connection on the origin side belongs to the CDN. The real client IP is passed as an HTTP header. On Cloudflare, the common header is the CF-Connecting-IP value. In many proxy chains, X-Forwarded-For is used, which typically contains a comma-separated list of IPs.

The critical point here is: X-Forwarded-For can also be sent by the client and can easily carry spoofed values on an origin server directly exposed to the internet. Therefore, the application or web server must only accept this header for connections coming from trusted proxy IP ranges. Otherwise, an attacker can make their own IP appear differently in the logs, polluting bot traffic analysis and security rules.

5-step plan for real IP logging

1. Define the trust boundary

First, clarify which networks your traffic reaches the origin from. You might be using Cloudflare, Fastly, Akamai, CloudFront, or your own Nginx proxy layer. If the origin server's ports 80 and 443 are directly open to everyone, someone can bypass the CDN and corrupt your logs with a fake X-Forwarded-For. If possible, allow traffic only from CDN IP ranges on your firewall.

2. Choose which header to use

If you are using Cloudflare, CF-Connecting-IP is a cleaner starting point. If you have multiple reverse proxy layers, you may need to use the X-Forwarded-For chain; in this case, a trusted proxy list is essential for the web server to select the real client from the correct position. Logging both the real IP and raw headers simultaneously helps you catch misconfigurations early.

3. Configure real IP settings on the web server

Nginx uses ngx_http_realip_module, and Apache uses mod_remoteip. The logic is the same: for requests coming from specific proxy IP ranges, the IP in the specified header is accepted as the client IP. In Nginx, after this process, $remote_addr can become the real client IP; it is useful to log the previous proxy IP separately using $realip_remote_addr.

4. Expand the log format for bot analysis

Standard access logs often provide only the URL, status code, and user-agent; however, this is insufficient for inspection behind a CDN. Log at least the following fields separately: ISO timestamp, real IP, CDN connection IP, raw X-Forwarded-For, CF-Connecting-IP, host, URI, status, response time, and user-agent. If you keep the fields with a fixed delimiter like a pipe, it becomes easy to parse them later with awk, a SIEM, or a log pipeline.

5. Measure IP count alongside user-agent

User-agent is not identity verification on its own; it can be spoofed. Nevertheless, it is useful for separating bot families for initial analysis. For example, the number of unique real IPs in requests appearing as GPTBot, the top 100 requested URLs, and IPs receiving 429 errors should be reported separately. For critical bots, checking the IP, ASN, or reverse DNS verification methods in official documentation provides more reliable results.

Technical checklist

• Test if the origin server can be accessed from outside the CDN; if it is open, close it with a firewall or security group.
• Fetch the current IP ranges of the CDN you use from the official source and introduce them to your web server as trusted proxies.
• Decide between CF-Connecting-IP for Cloudflare and X-Forwarded-For for general proxy chains based on your architecture.
• Add the real IP and CDN IP as separate fields to your log format.
• Log raw headers at least during a short transition period; this is how you will see incorrect headers or empty values.
• Assume that IP addresses may be considered personal data under GDPR/KVKK; determine your retention period, access authorization, and masking policy.
• Generate bot reports broken down by user-agent, real IP, URI pattern, status code, and request time.
• Check that your rate limit rule is working based on the verified real IP, not the CDN IP.

Example scenario: Logging with Nginx behind Cloudflare

The following example is for an Nginx origin using Cloudflare. The commands write the Cloudflare IP list to a separate file; if you are automating updates in production, enforce an nginx -t check in your test environment first.

sudo sh -c 'curl -s https://www.cloudflare.com/ips-v4 | sed s#^#set_real_ip_from # | sed s#$#;# > /etc/nginx/cloudflare-realip.conf'
sudo sh -c 'curl -s https://www.cloudflare.com/ips-v6 | sed s#^#set_real_ip_from # | sed s#$#;# >> /etc/nginx/cloudflare-realip.conf'

Define the real IP and custom log format in the Nginx server or http block:

include /etc/nginx/cloudflare-realip.conf;
real_ip_header CF-Connecting-IP;

log_format ai_bot '$time_iso8601|real_ip=$remote_addr|cdn_ip=$realip_remote_addr|xff=$http_x_forwarded_for|cfip=$http_cf_connecting_ip|status=$status|method=$request_method|host=$host|uri=$request_uri|rt=$request_time|ua=$http_user_agent';

access_log /var/log/nginx/ai-access.log ai_bot;

Test the configuration and reload Nginx:

sudo nginx -t
sudo systemctl reload nginx

Now you can measure how many unique real IPs you see for a specific AI bot user-agent:

grep 'GPTBot' /var/log/nginx/ai-access.log | awk -F'|real_ip=' '{print $2}' | awk -F'|' '{print $1}' | sort -u | wc -l

grep -E 'GPTBot|ClaudeBot|PerplexityBot|CCBot|ChatGPT-User' /var/log/nginx/ai-access.log | awk -F'|status=' '{print $2}' | awk -F'|' '{print $1}' | sort | uniq -c | sort -nr

The first command gives the number of unique real IPs for requests appearing as GPTBot. The second command counts which HTTP status codes the selected bot user-agents are returning. If you store these outputs on a daily basis, you can set thresholds for bot traffic analysis: for example, if a bot family usually comes from 20 IPs a day but jumps to 400 IPs in one night, you can investigate whether this is a content crawling strategy or fake user-agent traffic.

Common mistakes

• Trusting all X-Forwarded-For values: This header can be spoofed; accept it only when it comes from trusted proxy IPs.
• Rate limiting with the CDN IP: Thousands of real visitors may appear from the same CDN IP; the wrong threshold can affect legitimate traffic.
• Not logging raw headers at all: If the real IP setting is incorrect, retrospective diagnosis becomes difficult.
• Assuming user-agent is definitive identity: Every request saying GPTBot may not actually come from the relevant operator; use official verification signals as well.
• Storing IP data indefinitely: Determine a retention period based on analysis needs and keep access restricted.