How to spot AI bots in Apache/Nginx logs? Filter in 3 steps with 7 fields

Last week, a news site suddenly started showing hundreds of requests in the same minute; some behaved like a normal browser, while others were pulling page after page from a single user agent. At first, the site owner thought it was a "traffic spike," but once they looked at the Apache log, the pattern became much clearer: the same IP ranges, similar URIs, and unusual user-agent strings.

Short answer: To detect AI bots, first standardize Apache/Nginx access lines with 7 fields, then filter candidate bots from the user-agent, and finally verify with IP, request path, status code, and request frequency. A single user-agent line is not proof; spotting AI bots in server logs requires reading several signals together.

Why are 7 fields enough?

In raw logs, the goal is not to answer "which bot is definitely an AI bot" with one line. The goal is to build a record structure that lets you separate suspicious traffic in a few minutes. For that, these 7 fields are a solid starting point in most scenarios:

• IP address: Lets you group sequential requests from the same source.
• Timestamp: Shows traffic bursts and repetition intervals by minute.
• Request line: Shows which URI was called and with which method.
• Status code: Patterns like 200, 301, 404, 403, and 429 are very useful for filtering.
• Response size: Helps distinguish empty, short, or unusually large responses.
• Referer: Reveals traffic direction and source context.
• User-Agent: The fastest clue for building an initial candidate list.

It is important to read these 7 fields together, because user-agent alone can be spoofed, IP alone may sit behind NAT, and status code alone does not mean much. AI bot detection becomes clearer once these pieces are correlated within the same line.

How does filtering work in 3 steps?

1) Lock the log format into a single line

First, turn your Apache or Nginx access log into a format that is easy to parse. Using a pipe as the delimiter is practical; spaces inside timestamps and quoted fields won't get in the way. The examples below make all 7 fields directly reportable.

log_format ai7 '$remote_addr|$time_local|$request|$status|$body_bytes_sent|$http_referer|$http_user_agent';
access_log /var/log/nginx/access-ai7.log ai7;

LogFormat '%a|%t|%r|%>s|%b|%{Referer}i|%{User-agent}i' ai7
CustomLog logs/access-ai7.log ai7

At this stage, the goal is not to make the log "human-readable" but to make it "filterable from the command line."

2) Separate candidate bots by User-Agent

In the second step, search for AI bot strings that commonly appear in user-agent values. This list can change over time, so treat it as a starting filter, not as a fixed truth. In examples, strings like GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, CCBot, Amazonbot, Bytespider, and Applebot-Extended often show up.

grep -Ei 'GPTBot|OAI-SearchBot|ClaudeBot|PerplexityBot|CCBot|Bytespider|Amazonbot|Applebot-Extended' access-ai7.log

The critical point here is that a user-agent match is not enough on its own. A bot with the same name can behave differently, and malicious clients can also mimic these strings. So the output of the second step is a "candidate list," not a final verdict.

3) Verify with IP, request path, and status code

In the third step, you test candidate lines more strictly. Is the same IP sending hundreds of requests in a short time? Is it walking the same directory structure? Is it returning only 200s, or is it also producing 404s and 429s? Is the referer field empty, or is there a natural transition from inside the site? The answers to these questions make the difference between an AI bot and normal traffic visible in the server log.

In practice, these three patterns are especially useful:

1. High repetition: Many GET requests in a short time from the same IP and user-agent.
2. Path pattern: Back-to-back access to /robots.txt, /sitemap.xml, or similar content clusters.
3. Status code trail: 301 redirects, 404 missing pages, or 403/429 rate-limit responses.

This combination ties the user-agent AI detection signal to bot traffic analysis and gives you a fast first-pass filter.

When does misclassification happen?

The biggest mistake is treating every suspicious line as an AI bot. For example, real users behind a corporate proxy can generate many requests from the same IP. A CDN, health-check service, or uptime tool can leave a similar trail as well. That is why you should not decide based on a single field; look at the combined pattern across all 7 fields.

Another risk is doing bulk counts without considering timestamps. 300 requests per day may look normal; the same 300 requests arriving within 2 minutes tell a very different story. So AI bot detection is more about speed and repetition patterns than raw totals.

Technical checklist

• Define a 7-field, single-delimiter log format for Apache or Nginx.
• Keep at least several days of logs; one hour of data hides most patterns.
• Use user-agent strings as a candidate filter, not as the final decision.
• Check repetition frequency using the IP + time + URI combination.
• Think of 404, 403, 429, and 301/302 ratios as separate columns.
• Inspect the difference between a blank referer and a natural navigation path.
• Create an allowlist and a test copy before blocking suspicious bots.
• Back up the raw logs before every change; you need to be able to roll the filter back.

Example scenario: real filtering with 7 fields

Imagine you have an Nginx log in this format: IP, time, request, status, size, referer, user-agent. First pull the AI bot candidates, then count only specific IPs, and finally look at which paths were requested.

awk -F'|' 'BEGIN{IGNORECASE=1} $7 ~ /(GPTBot|OAI-SearchBot|ClaudeBot|PerplexityBot|CCBot|Bytespider|Amazonbot|Applebot-Extended)/ {print}' access-ai7.log

awk -F'|' 'BEGIN{IGNORECASE=1} $7 ~ /(GPTBot|OAI-SearchBot|ClaudeBot|PerplexityBot|CCBot|Bytespider|Amazonbot|Applebot-Extended)/ {c[$1]++} END {for (ip in c) print c[ip], ip}' access-ai7.log | sort -nr

awk -F'|' '$4 ~ /^(301|302|403|404|429)$/ {print $0}' access-ai7.log | head -n 50

With these three commands, you first see the candidates, then understand which IP is concentrating the traffic, and finally evaluate which responses the bot triggered. If the same IP is moving through many URIs in a short time with the same user-agent and the referer remains empty, your record is no longer just "traffic"; it becomes an analyzable bot trace.

Common mistakes

• Making a decision based only on user-agent.
• Trying to filter before standardizing the log format.
• Ignoring response codes and request frequency.
• Generalizing from a single day of data.
• Assuming legitimate traffic from a proxy/CDN is a bot.

Conclusion

AI bot detection is not a mysterious reverse-engineering task; it is about clean logs, simple filters, and consistent verification. Once you capture Apache/Nginx logs with 7 fields, you get a fast starting point for user-agent AI detection; when you combine that with IP, time, request, and status codes, spotting AI bots in server logs becomes much easier to read. In short: find the candidates first, verify the behavior next, and act only after that.