AI bot detection is the technical process of assessing whether a web request was sent by an artificial-intelligence crawler or an automated bot based on technical signals. When requests from GPTBot, OAI-SearchBot, or another crawler start appearing in the server logs of an e-commerce site, the real issue is not only identifying the bot; IP address, user agent, requested URL, and timing information can also be stored in an uncontrolled way.
Short answer: do not retain AI bot logs indefinitely with raw IPs and unnecessary URL parameters. At the point closest to the source, mask the IP at least to /24 for IPv4 and /64 for IPv6; keep bot classification signals in separate fields; apply a 30-day sample retention policy together with purpose, access, and deletion records. From a KVKK perspective, 30 days is not an automatic requirement for every business; the period should be justified by the stated purpose and the necessary retention need.
Which log fields should be kept after AI bot detection?
Keep only the fields needed for bot traffic analysis: UTC timestamp, normalized request path, HTTP status code, request method, response size, user agent, and bot classification result. Do not log raw IP, full query string, cookies, authorization headers, or referrer fields that contain personal data by default.
It is not correct to say that a bot request truly belongs to a specific AI company based on the User-Agent alone. User-Agent can be spoofed; IP range, reverse DNS, request behavior, and the publisher's documented crawler information should be reviewed together. Storing the result in a classification field such as ai_crawler_candidate, verified_crawler, or unknown_bot makes the evidence level visible instead of claiming certainty.
What does IP masking provide under KVKK?
IP masking reduces direct identifiability; however, by itself it does not guarantee anonymization. If a masked IP can still identify a person when combined with time, user agent, and other systems, the data may still fall within the scope of personal data.
As a practical starting policy, you can drop the last octet of an IPv4 address and store it as 192.0.2.0/24, and store IPv6 with a /64 network prefix. For example, 192.0.2.123 is recorded as 192.0.2.0/24. But these levels do not automatically mean anonymization for every system. Re-identification may still be possible because of a small network, a limited number of visitors, or detailed time information.
Generating a fixed SHA-256 digest from the raw IP is also not automatically anonymization. Because the IP space is limited and predictable, dictionary attacks are possible. If you need to correlate bots within the same source network, use keyed HMAC-SHA-256 on the masked value. This makes matching difficult without the key; even so, this method is generally considered pseudonymization rather than anonymization.
How should a 30-day log retention period be implemented?
Calculate the 30-day period starting from the UTC time the log was created, and trigger automatic deletion or irreversible anonymization once the period expires. Justify this period with incident review, bot abuse, and security monitoring needs; do not set it merely because it is “the industry standard.”
KVKK does not set a universal 30-day period for every AI crawler log. The data controller should assess the processing purpose, legal basis, proportionality, data subject risk, and any retention obligation arising from another regulation. If records need to be moved to a separate incident file in exceptional cases such as a security event, the incident number, the people who accessed it, and the new deletion date should be documented in writing. This exception should not turn into indefinite retention of all logs.
Example log retention policy
- Purpose: Classify AI crawler behavior and review anomalous bot traffic and security incidents.
- Scope: Access logs generated in CDN, WAF, reverse proxy, application, and bot analysis systems.
- Data minimization: /24 masking for IPv4 and /64 masking for IPv6; exclude query strings and cookies by default.
- Retention: Standard bot logs for 30 days; expired records are removed within 24 hours by a daily deletion job.
- Exception: Records transferred to an opened security incident are kept separately, with the reason and new final deletion date noted in the incident record.
- Transparency: The privacy notice explains server security, abuse prevention, data categories processed, and the retention approach.
What are the 6 access rules for AI crawler security?
The following 6 rules move log access away from the “any technical team member can read them” approach and place it on a foundation of role, purpose, and auditability. The rules should cover both raw source logs and derived logs with masked IPs.
- Role-based access: Only the security manager whose duties require it and a limited number of system administrators should access raw or detailed security logs. SEO, content, or marketing teams should receive only aggregated bot reports.
- Multi-factor authentication: MFA should be mandatory for log repositories, SIEM dashboards, and cloud storage accounts. Shared admin accounts should not be used; each access should be made with a personal account.
- Separate raw and derived data: Keep raw source logs in a separate repository and masked, bot-labeled reports in another. Remove direct analyst access to the raw repository.
- Purpose-limited queries: Access should be for bot classification, security review, or system error analysis. Curiosity-driven queries about personnel, customers, or visitors should be prohibited and clearly stated in the procedure.
- Access logging: Record who accessed which repository, on what date, and with which incident or request number. Access audit logs should be protected separately from the bot logs being reviewed and must not be alterable by unauthorized persons.
- Export controls: CSV, screenshots, or local file export should be disabled by default. For exceptional exports, record approval, purpose, recipient, file encryption, and deletion date.
How is AI bot data masked in server logs?
If masking is done in application code while the raw access log remains on the server, the risk is not eliminated. Perform the transformation at the CDN, WAF, reverse proxy, or log agent layer before the data reaches central storage. If the source system must create a raw log, restrict file permissions, store it encrypted, and delete it after a short technical buffer period.
The Python example below masks IPv4 at the /24 level and IPv6 at the /64 level; it then generates a keyed HMAC over the masked value. Do not hard-code the HMAC key into the code; store it in a secret variable or key management service.
from ipaddress import ip_address, ip_network
import hashlib
import hmac
import os
def mask_ip(value):
try:
ip = ip_address(value.strip())
except ValueError:
return None
prefix = 24 if ip.version == 4 else 64
network = ip_network(f'{ip}/{prefix}', strict=False)
return f'{network.network_address}/{prefix}'
def stable_token(value, secret):
if not value:
return None
return hmac.new(secret, value.encode('utf-8'), hashlib.sha256).hexdigest()
secret = os.environ['LOG_HMAC_KEY'].encode('utf-8')
masked = mask_ip(source_ip)
record = {
'ip': masked,
'ip_token': stable_token(masked, secret),
'bot_class': bot_class,
'status': http_status
}Make sure this code does not write the source_ip variable to the raw file. Also check the reverse proxy's own access log format; even if the application masks later, Nginx, Apache, CDN, or WAF may still write the raw IP to a separate channel. Keeping the token stable enables correlation, but token access should also be subject to authorization and retention rules.
Practical technical checklist
- Do not use User-Agent as the sole evidence in AI bot classification; report the “unknown” state separately.
- Remove query strings, cookies, Authorization headers, and unnecessary referrer values from the log format.
- Test the re-identification risk of /24 masking for IPv4 and /64 masking for IPv6 in the context of your organization.
- List retention periods separately for CDN, WAF, reverse proxy, application, and central log stores.
- Verify the 30-day deletion job in a test environment; document how long deleted data remains in backups.
- Write the 6 access rules into the procedure; review the permission list and access records at least once a year.
- Keep the privacy notice, data inventory, retention-deletion policy, and security incident procedure consistent with one another.
Example scenario: a 30-day policy on a content site
A content site receiving around 1,500 access logs per day classifies GPTBot-like User-Agents as “candidate crawlers.” The site separately checks actual crawler verification using the publisher's current documentation, request behavior, and network information; it never treats a record as verified solely because of the User-Agent.
In the setup, query strings are removed at the CDN layer, the reverse proxy log stores IP as /24 or /64, and only UTC time, path, status code, response size, classification label, and masked IP are sent to central storage. The security team performs daily bot traffic analysis for 30 days. A task running at 02:00 every day deletes expired records; the deletion result and any error information are written to a separate audit log.
If a suspicion of attack arises, the security owner opens an incident number, justifies the relevant time range and data fields, limits access to 2 authorized people, and adds the new deletion date to the incident record. The marketing team, on the other hand, can only see daily total crawler requests and status code distribution; it cannot see IP- or request-level records. This structure addresses AI crawler security needs together with personal data minimization and auditable access.
Common mistakes
- Assigning a definitive identity when seeing “GPTBot” or another User-Agent without verifying the source.
- Assuming a hashed IP is automatically anonymous data.
- Masking the IP in the application but forgetting the web server's raw access log.
- Extending 30 days to 1 year or indefinitely without justification.
- Thinking of robots.txt as access control; robots.txt does not technically block an unauthorized bot's request.
- Using a shared account for the log panel and not tracking who reviewed which record.