llms.txt Validation Guide: 10 Checks for 404, 301, Canonical, and Noindex

On a Monday morning, the content team published a new llms.txt file; it listed product documentation, pricing details, and top guides. However, testing showed that https://site.com/llms.txt first redirected via 301 to a language selector and then returned a 404. Seeing the file in a browser wasn't enough; the HTTP signals seen by an AI agent were different.

The short answer: To validate a live llms.txt file, you must measure that the root URL returns a 200 status, doesn't create unnecessary 301 chains, doesn't send incorrect canonicals or X-Robots-Tag: noindex headers, and that the links within the file are accessible. The question of how to write an llms.txt file is about content templates; this guide tests whether the file is technically readable by AI agents.

What does llms.txt validation measure?

llms.txt is a text-based file recommended for publication in the root directory to summarize important resources on your site for AI systems. Think of it as an AI documentation file: product pages, technical docs, policy texts, and evergreen reference content are presented with organized headers. However, this file is not an official search engine standard; different bots may read it at different frequencies or not at all.

Therefore, validation translates your intent into measurable web signals. What does the AI agent see when it requests the URL? A 404 or a 200? Is the CDN converting the file into an HTML error page? Is a noindex header left over from the staging environment still being sent? Do the URLs inside the file pass through 301 chains? These can be tested in minutes from the command line.

HTTP status: 200 or invisible error?

The first check is the status code of the full URL: https://yourdomain.com/llms.txt. The expected result is usually 200 OK and text-based content. 404 indicates the file is not found, 403 indicates access is blocked, and 5xx indicates a server or CDN-side issue. Some sites show the file when logged in as an administrator, but return a 403 for anonymous requests; AI agents almost always make anonymous requests.

Another practical test is file size. An empty file or one with two lines might technically return a 200 but doesn't provide a useful signal for the agent. Conversely, a massive list containing hundreds of thousands of URLs tries to act like a sitemap for AI and leads to loss of focus. llms.txt does not replace an XML sitemap; it is more useful as a curated and explained list of resources.

301 and redirect chains: there is tolerance, but there is a cost

A single 301 redirect from HTTP to HTTPS is standard for most clients. The problem arises with chains like /llms.txt going to /llms.txt/, then to /en/llms.txt, and finally to the homepage. Some bots follow redirects, while others may fail to reach the final content due to timeouts, policies, or limits. The cleanest structure is to return a 200 directly at the canonical address you publish.

Also, watch for case sensitivity. On Linux-based servers, /LLMS.txt and /llms.txt might be different files. If you have written the URL in uppercase in your documentation, footer link, or developer page, check both in your test and redirect them to a single address.

Canonical and noindex: the signal is in the header for text files

In HTML pages, canonical is usually provided via a rel=canonical tag. When llms.txt is served as plain text, there is no in-page HTML canonical; if present, it comes via an HTTP Link header. If the header points to the https://yourdomain.com/ homepage as canonical, you are pointing to another URL, not the file itself. If you are not using a canonical, that is fine; if you are, it must be consistent with the final llms.txt URL.

On the noindex side, the critical signal is the X-Robots-Tag header. Configurations that globally add noindex, nofollow, especially in staging, test, or membership areas, can be accidentally applied to the llms.txt file. Noindex does not mean a reading ban for every bot; reading blocks are provided via robots.txt. However, it is a signal that could weaken the discoverability of the file for AI search engines and classic search systems.

Technical checklist: 10 validation points

1. Root URL test: The https://yourdomain.com/llms.txt address must return 200 for an anonymous request.
2. 404 and 403 test: Check if different status codes are generated for mobile, desktop, and bot-like User-Agents.
3. 301 chain: Measure how many steps it takes to reach the final URL; if possible, return a 200 directly at the published HTTPS address.
4. Content-Type: Send text/plain or text/markdown and preferably charset=utf-8; ensure no HTML error template is returned.
5. Canonical header: If an HTTP Link canonical exists, it must point to the final llms.txt URL; it should not point to the homepage or a category.
6. Noindex header: Check that X-Robots-Tag: noindex or none is not being sent.
7. robots.txt access: Verify that there is no Disallow rule blocking the /llms.txt path for the AI bots you are targeting.
8. In-file links: Test that every URL within llms.txt returns a 200 and does not lead to a login wall or a soft 404 page.
9. Link canonicals: Important pages listed in the file should point to themselves or the correct canonical URL; do not use parameterized copies.
10. Cache and freshness: Check that the new version arrives after a CDN purge and that Last-Modified or ETag signals represent the expected file.

Example scenario: testing the live file in 5 minutes

In the example below, our domain is example.com. You can run the same commands with your own domain. The goal is to see the headers first, then download the file and check the URLs inside one by one for their status codes.

DOMAIN=example.com
curl -sS -I https://$DOMAIN/llms.txt
curl -sS -L -D headers.txt -o llms.txt https://$DOMAIN/llms.txt
grep -iE 'HTTP/|location:|content-type:|x-robots-tag:|link:|cache-control:' headers.txt
head -40 llms.txt
grep -Eo 'https?://[^ )]+' llms.txt | sort -u > llms-links.txt
while read u; do code=$(curl -sS -o /dev/null -w '%{http_code}' -L $u); printf '%s %sn' $code $u; done < llms-links.txt

In an ideal output, the final response is 200, the content type is text, there is no noindex header, and the critical URLs inside the file are 200. Seeing a 301 in the output is not an error by itself; the error is if the chain ends at the homepage, language selector, or a 404 page. If you see X-Robots-Tag: noindex, find where this header is added in your web server, CDN rule, or application middleware layer.

For a simple robots.txt check, look for these lines. The following example is an undesirable configuration because it blocks the llms.txt path for all bots:

User-agent: *
Disallow: /llms.txt

Instead, if you use a general block, you can define an explicit exception for llms.txt. Since robots.txt interpretation rules can vary by bot, you should test with the specific User-Agent you are targeting in the live environment.

Common mistakes

• Embedding the file only in the sitemap: The expected practice for llms.txt is a readable URL in the root directory; a sitemap reference alone may not be sufficient.
• Returning an HTML page: If an SPA redirect or 404 template returns a 200, the bot reads the application shell instead of the text file.
• Forgetting the staging noindex header: The global noindex header may remain in the file after going live.
• Listing the entire site: Instead of thousands of URLs, select decision-making content, documents, and trusted reference pages.
• Using parameterized URLs: Addresses with UTMs or filters create canonical conflicts and unnecessary redirects.
• Testing only once: Repeat the same check after every deployment as CDN, deployment, and security rules change.