llms.txt URL prioritization is the process of selecting, with measurable rules, the most valuable URLs to present to AI systems from among hundreds of web pages. Imagine an e-commerce site with 500 indexable pages, but only 70 of those pages actually contain up-to-date product information and strong purchase context. The short answer: rather than piling every URL into llms.txt, assign each page five scores from 0-5, calculate a weighted total, and in the first pilot validate the top 50-100 URLs editorially.
llms.txt is a Markdown-based file, usually served from the site root, intended to explain important resources to AI tools. It does not block access like robots.txt, does not have to list every URL like an XML sitemap, and it is not guaranteed to be read by every AI tool. For that reason, the issue is less about the file’s format and more about choosing the right sources.
Quick answer: which URLs should be selected from 500?
In your selection model, you can weight freshness at 20%, commercial value at 25%, content completeness at 25%, search intent and answerability at 15%, and uniqueness plus accessibility at 15%. Convert the 0-5 scores into a 100-point scale by applying the weights. Treat pages scoring 70 and above as candidates, 50-69 as review queue items, and anything below 50 as excluded by default.
Why is llms.txt URL selection an editorial scoring task?
The fact that a URL exists on the site does not mean it is worth using in AI responses. An old campaign page, a parameterized filter URL, 12 category pages repeating the same text, or an out-of-stock product page can make the file noisy. Thinking of llms.txt as a site map for AI is a useful analogy, but using it as a complete technical sitemap is not correct.
First gather the URL pool from the CMS export, XML sitemap, and analytics tool. Then deduplicate the URLs; keep public pages that return 200 HTTP status, point to the correct canonical target, and have clear user value on the candidate list. Parameterized, redirected, login-required, or removed URLs should remain in a separate review group.
How should the five scoring rules be applied?
1. How is freshness scored?
Score freshness not only by the date shown on the page, but by price, stock, regulations, product features, and the date the sources were actually checked. In a sample rubric, the last 30 days gets 5 points, 31-90 days gets 4, 91-180 days gets 3, 181-365 days gets 2, 366-730 days gets 1, and content older than 730 days gets 0 points. For news or regulatory pages, the thresholds can be stricter; for evergreen guides, more flexible.
Automatically updating the date field should not artificially inflate the score. Even if the main claim has not changed, sections such as price, warranty terms, or technical compatibility may still be outdated. The editor should match the last updated date against actual source changes. A URL with 5 points should still be excluded if it contains incorrect information.
2. How is commercial value measured?
Commercial value is a page’s potential to generate revenue, leads, registrations, or qualified demand directly or indirectly. 0 points can be given to a page that only builds general awareness and leads nowhere; 3 points to a page with a clear solution or comparison; 5 points to a page tied to purchase, quote requests, demos, or a high-margin service.
To avoid relying on intuition, use form submissions, phone clicks, add-to-cart actions, or sales relationships from the last 90 days. Not every high-traffic page is commercially valuable; for example, a dictionary page may attract many visitors and generate no qualified demand. A product page may have high commercial value, but if stock, pricing, and delivery information are missing, the content completeness score should drop.
3. What tests should content completeness pass?
Content completeness measures whether a page can answer the question on its own and support its key claims. Use a practical five-item test: is the target audience and topic clear, is the main answer given early, are technical or commercial conditions stated, is a source or proof provided, and is there a logical next step for the reader? Give 1 point for each yes.
If a page consists only of headings, short teaser sentences, or links to other URLs, it should score low. On a product comparison page, failing to define criteria, on a service page failing to state scope and exclusions, or in a guide lacking current sources are completeness issues. If an AI system uses the page, missing context can turn into an incorrect or overly generic answer.
4. How are search intent and answerability scored?
Measure how directly the page answers one main question or task. 5 points can be given to a page whose title, subheadings, and examples all align with the same user intent; 3 points to a page that reasonably covers multiple intents; 0 points to a page with an unclear title or one that mixes unrelated topics. If Search Console queries are available, tie real queries into this assessment.
Rather than a URL superficially targeting five different keywords, it is better for it to answer one primary question in depth. For example, a shipping integration guide should cover supported systems, setup steps, limitations, and cost items in the same context. This is more informative than looking only at traffic volume when selecting important pages.
5. How is uniqueness and accessibility verified?
The final score checks whether the page is not a copy of another URL and whether it can be fetched by machines. 5 points can be given to a page that returns 200 status, declares the correct canonical, is unique in content, and is linked internally. Duplicate text, broken links, incorrect canonical, or URLs created only through filter parameters should be rated in the 0-2 range.
This rule carefully separates variants on commercial pages. If color or size parameters repeat the same description, choose the main product URL instead of adding them all to llms.txt. Variants that truly carry different technical features and user intent can be separate candidates. The URL added to the list should be the final page accessible without login.
How do you calculate the weighted score?
Enter the five scores for each URL into a spreadsheet. Let freshness be F, commercial value C, completeness K, intent I, and uniqueness U. To convert the 0-5 scores to a 100-point scale, you can use this formula:
=ROUND((F2*20+C2*25+K2*25+I2*15+U2*15)/5,0)For example, a product page with F=4, C=5, K=4, I=5, and U=5 gets 88 points. Another page with F=1, C=4, K=2, I=3, and U=5 stays at 57 points. These thresholds are not a universal AI rule; they should be adjusted to your editorial capacity and business goals. If you can update 20 pages per month, prioritize the high-scoring candidates, but do not exclude pages that are legally or product-safety critical just because of the score.
Technical checklist
- Combine all URLs from the CMS, XML sitemap, and analytics data into a single table.
- Normalize URLs; check differences in http/https, trailing slash, case, and parameters.
- For each candidate, record the last real content update, HTTP status, and canonical target.
- Score the five rules from 0-5 and calculate the weighted total automatically.
- Validate candidates scoring 70 and above with an editor; improve commercial pages with low completeness scores first.
- Create a controlled list with 50-100 URLs in the first version; do not bloat the file to cover all 500 pages.
- Test that every selected URL is public, current, and a final address that answers a real user question.
- Publish the file at /llms.txt in the site root and track changes with version control or the CMS log.
Example scenario: a 500-page B2B software site
A B2B software company crawls 500 URLs and finds 180 blog posts, 120 integration pages, 80 feature pages, 60 customer stories, and 60 support pages. It first separates redirects, duplicate tag pages, and 90 URLs that require access from the candidate list. The remaining pages are scored against the five rules. The highest averages appear on current integration, pricing, security, feature, and use-case pages.
The editor does not add a security page scoring 82 that was updated 8 months ago directly; instead, they first refresh the certificate and data processing information. A blog post scoring 74 is moved to the review group because its commercial connection is weak. The final list contains 64 URLs that explain product and service context. A sample llms.txt file could look like this:
# Acme Yazılım
> Integration, security, and usage resources for B2B teams.
## Product and Pricing
- [Platform features](/platform/ozellikler)
- [Pricing](/fiyatlandirma)
## Technical Resources
- [API integration guide](/gelistiriciler/api-rehberi)
- [Security and data processing](/guvenlik)
## Use Cases
- [E-commerce integration](/senaryolar/e-ticaret)This example shows selected resources grouped by topic instead of automatically dumping every URL into the file. Being included in the file does not mean any AI tool will definitely crawl, use, or cite that page. The value lies in creating a cleaner editorial signal that makes the right pages visible in a disciplined way.
Common mistakes
- Thinking llms.txt is robots.txt and that it grants or blocks crawl permission.
- Turning the file into a second sitemap by listing all 500 URLs.
- Scoring freshness only by publication date.
- Adding high-value but incomplete product and service pages without content review.
- Listing parameterized, duplicate, or redirected URLs instead of the canonical target.
- Publishing the list once and never tracking pricing, stock, regulations, or product changes.