Migrating from XML Sitemap to llms.txt: A 4-Step Automation Setup

When an AI agent or a Large Language Model (LLM) visits your website, it can read traditional XML sitemaps, but these files are essentially hierarchical data dumps designed for machines. What AI agents truly need is a human-readable, Markdown-based guide that explains the context, provides brief summaries, and highlights which pages are critical. This is exactly where llms.txt comes into play.

The Short Answer: What is llms.txt and Why Automate It?

llms.txt is a Markdown file located in your website's root directory that provides AI models with your site's structure, key pages, and short descriptions. Automatically converting the URLs from your XML sitemap into this new format ensures that your content updates are instantly and accurately perceived by AI systems (Perplexity, GPT-4o, Claude, etc.). This saves you from the hassle of manual file updates and maintains your AI-SEO compatibility.

Why Are Traditional XML Sitemaps Not Enough?

XML sitemaps (sitemap.xml) are perfect for crawlers like Googlebot to discover URLs. However, these files do not provide semantic information about the content. They only convey that 'this URL exists' and 'it was last updated on this date.' An LLM, on the other hand, wants to know what a page is about before visiting it. The llms.txt file helps the LLM use its token budget efficiently by adding a short description for each link, allowing it to crawl only relevant pages.

4 Steps to Automate the Setup from XML to llms.txt

1. Identifying and Filtering the Data Source

The first step of automation is to use your existing sitemap.xml file as a data source. However, adding every URL to your llms.txt file is a mistake. For AI agents, you should only select pages with 'high informational value.' Your filtering criteria should be:

• Live pages returning an HTTP 200 status code.
• Pages that do not contain a noindex tag.
• Informational pages, excluding 'transactional' pages like carts, checkouts, or user profiles.
• Pages where the canonical tag points to itself.

2. Extracting Meta Data (Scraping or API)

XML files only contain URLs. For llms.txt, you need to add a short description next to each URL. You have two ways to generate these descriptions automatically:

• SEO Meta Description: You can use your existing meta descriptions.
• AI Summarization: You can use a Python script to fetch the first 200 words of each page and generate single-sentence summaries using a cost-effective model like gpt-3.5-turbo.

3. Converting to Markdown Format

You need to format the URL and description pairs according to the llms.txt specification. The standard structure is as follows:

# Site Title
> A short description of the site's general purpose.

## Main Section (e.g., Blog)
- [Page Title](URL): A short summary of the page.
- [Other Page](URL): Summary of the other page.

4. Deploying via CI/CD or Cron Job

Instead of manually uploading this Markdown file to the server, you should set up a GitHub Action or a server-side Cron Job. This task should check your XML sitemap every night, detect newly added pages, and update your /llms.txt file in the root directory.

Technical Checklist

• File Location: Ensure the file is accessible at domain.com/llms.txt.
• File Size: Do not keep the file too large to avoid straining the LLMs' context window. If you have too many pages, create a second file named llms-full.txt and link to it from the main file.
• Character Encoding: Always use UTF-8.
• Markdown Standards: Use # for headers and - for lists correctly.
• Access Permissions: Ensure your robots.txt file does not block this file.

Example Automation Scenario (Python)

The following simple Python logic can help you read an XML sitemap and create a basic llms.txt draft:

import requests
import xml.etree.ElementTree as ET

def generate_llms_txt(sitemap_url):
    response = requests.get(sitemap_url)
    root = ET.fromstring(response.content)
    
    with open('llms.txt', 'w', encoding='utf-8') as f:
        f.write("# My Website's AI Guide")
        f.write("> This file is a content index for AI agents.nn")
        f.write("## Featured Contentn")
        
        for url in root.findall('.//{http://www.sitemaps.org/schemas/sitemap/0.9}loc'):
            # A function to fetch meta descriptions for each URL can be called here
            f.write(f"- [{url.text}]({url.text}): Page description will go here.n")

generate_llms_txt('https://example.com/sitemap.xml')

Common Mistakes

• Adding All URLs: Adding all products from a 50,000-page e-commerce site to llms.txt will confuse the AI model. Only include category and guide pages.
• Leaving Descriptions Empty: An llms.txt consisting only of links is no different from an XML sitemap and holds low value for AI.
• Forgetting to Update: Leaving deleted pages in the llms.txt file causes AI agents to encounter 404 errors.