Identifying Fake AI Bots: 4-Step IP Verification via Reverse DNS Lookup

When reviewing your server logs, you might encounter a visitor making dozens of requests per second, claiming to be 'GPTBot' or 'Googlebot'. However, if your server load is significantly higher than usual and your bandwidth is depleting rapidly, trusting only the 'User-Agent' header is like opening your door to anyone who claims to be a 'police officer' without asking for identification.

Short Answer: Checking only the User-Agent string to verify a bot's identity is insufficient, as this value can be spoofed in seconds via HTTP headers. For definitive verification, you must perform a Reverse DNS (rDNS) lookup to confirm that the IP address resolves to a hostname belonging to the claimed organization (OpenAI, Google, Microsoft, etc.), and then verify that this hostname resolves back to the same IP (Forward DNS).

Why You Shouldn't Rely Only on User-Agent Checks

In the HTTP protocol, the User-Agent is a text string where the client identifies itself to the server. Changing this string to 'Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)' using any Python script or a simple cURL command is technically a 5-second task. Malicious bots (scrapers) frequently use this method to bypass firewalls and extract data by masquerading as 'good bots'. Unlike real bots, fake bots do not follow robots.txt rules, ignore crawl-delay limits, and consume your server resources, slowing down access for real users.

Step 1: Identifying the IP Address and Initial Lookup

The first step in the verification process is capturing the IP address from which the suspicious request originated. Find the relevant IP in your Nginx or Apache logs by searching for the 'GPTBot' string. For example, let's assume the suspicious IP is 20.15.240.12. To determine if this IP truly belongs to OpenAI, you should use the host or nslookup command in your terminal.

Running a Reverse DNS (PTR) Lookup

Type the following command into your terminal:

host 20.15.240.12

If this IP is a genuine OpenAI bot, the output should look like this: 12.240.15.20.in-addr.arpa domain name pointer gptbot-20-15-240-12.openai.com. The critical point here is that the domain name ends with openai.com. If the result shows amazonaws.com, digitalocean.com, or a random ISP name, it is a case of a fake bot user-agent.

Step 2: Cross-Verification with Forward DNS

Performing only an rDNS lookup can sometimes be misleading, as advanced attackers can create deceptive PTR records on their own DNS servers. Therefore, you should implement a method called 'Double-Reverse DNS'. Resolve the hostname you obtained from the first query back into an IP address:

host gptbot-20-15-240-12.openai.com

The IP address obtained from this query must match the 20.15.240.12 address you saw in your logs exactly. If it does not match, this is an attempt at 'DNS spoofing' or impersonation.

Step 3: Verification Criteria for Popular AI Bots

Every major tech company uses specific domain name patterns to verify their bots. Here are the most common ones:

• OpenAI (GPTBot, ChatGPT-User): The rDNS result must end with openai.com.
• Google (Googlebot, Google-InspectionTool): The rDNS result must end with googlebot.com or google.com.
• Microsoft (Bingbot, BingPreview): The rDNS result must end with search.msn.com.
• Perplexity: Usually uses AWS or its own infrastructure, but they publish official IP lists.

Step 4: Automated Verification and Firewall Configuration

Checking every request manually is impossible. To automate the AI bot detection process, you should define rules at the server level or on a CDN (like Cloudflare). For sites using Cloudflare, the 'Verified Bot' feature performs these rDNS checks in the background automatically and blocks fake bots instantly.

Verification with a Sample Python Script

The following simple Python code provides basic logic to check if an IP address and User-Agent are consistent:

import socket

def verify_bot(ip, claimed_domain):
    try:
        # Reverse DNS
        hostname, _, _ = socket.gethostbyaddr(ip)
        if not hostname.endswith(claimed_domain):
            return False
        
        # Forward DNS
        resolved_ip = socket.gethostbyname(hostname)
        return resolved_ip == ip
    except (socket.herror, socket.gaierror):
        return False

# Usage: verify_bot("20.15.240.12", "openai.com")

Technical Checklist

• Extract suspicious User-Agent and IP pairs from log files.
• Perform an rDNS lookup on the IP address using the host command.
• Ensure the returned hostname ends with the official domain (openai.com, googlebot.com, etc.).
• Resolve the hostname back to an IP (Forward DNS) and compare it with the original IP.
• Permanently block IPs confirmed as fake via iptables or Cloudflare WAF.
• Regularly update the IP ranges of legitimate bots (provided in JSON format).

Common Mistakes

• Trusting Only the User-Agent: This is the biggest mistake; 90% of bot traffic can be easily manipulated.
• Blocking All Data Center IPs: Some legitimate services (e.g., uptime monitors) also use AWS or Azure. Blocking them without performing rDNS can cut off useful traffic.
• Not Clearing DNS Cache: Relying on old DNS records during IP changes can lead to incorrect blocks.