Automating llms.txt: 3 Script Methods for Dynamic Websites

Imagine you have an e-commerce site with thousands of products or a news portal that updates every hour. Manually preparing an llms.txt file means dealing with a static document that becomes obsolete in just a few hours. When AI agents (GPTBot, Claude-Bot, etc.) visit your site, they want to find the most up-to-date data in the simplest format; this makes automation a necessity.

Short Answer: How to Write llms.txt Automatically?

The technical answer to the question how to write llms.txt for dynamic websites is to use a server-side script to convert database queries into Markdown format. The core logic is to create a 'route' that serves a structure containing your site's main headings, critical pages, and short summaries at the /llms.txt path with the Content-Type: text/plain header.

Why Automated llms.txt Instead of Manual?

Traditional sitemaps (XML) are designed for search engine indexing processes. However, LLMs (Large Language Models) need context to make sense of data. A manually prepared file carries the following risks:

• Data Staleness: If a newly added category or article is not in the file, the AI agent may act as if that information does not exist.
• Loss of Context: Manually writing a summary for every page in static files is an operational burden.
• Scaling Issue: On a site with 10,000+ pages, manually selecting which pages are 'priority' increases the margin for error.

Method 1: Dynamic Generation with PHP and Laravel

In PHP-based systems (including WordPress), you can generate real-time output by pulling the most popular or recent content from the database using a controller. This method is the most efficient, especially for content management systems.

Example PHP Script Logic

// routes/web.php
Route::get('/llms.txt', [LLMController::class, 'generate']);

// LLMController.php
public function generate() {
    $pages = Page::where('is_active', true)->limit(50)->get();
    $output = "# Your Site Namen> Short description and purpose of the site.nn";
    
    foreach($pages as $page) {
        $output .= "- [{$page->title}]({$page->url}): {$page->excerpt}n";
    }
    
    return response($output)->header('Content-Type', 'text/plain');
}

This code snippet always presents the 50 most current pages in the database as a Markdown list to the AI bot visiting your site. Using the text/plain header ensures the bot perceives the file as raw text rather than an HTML page.

Method 2: Middleware Solution with Node.js and Express.js

In modern JavaScript-based web applications, defining the llms.txt file as a middleware or a custom endpoint is quite simple. The speed of Node.js allows you to format large datasets in seconds.

Express.js Implementation

app.get('/llms.txt', async (req, res) => {
    const data = await db.collection('posts').find().limit(20).toArray();
    let markdown = "# Knowledge Hubnn";
    
    data.forEach(item => {
        markdown += `## ${item.title}n- URL: ${item.permalink}n- Summary: ${item.summary}nn`;
    });
    
    res.set('Content-Type', 'text/plain');
    res.send(markdown);
});

The point to note here is the output size. If you are generating thousands of lines, you should keep only the most important links in the main llms.txt file and provide a link to an llms-full.txt file for details. AI models usually prioritize the first 100KB of data.

Method 3: Markdown Conversion with Python and Django/Flask

Python is one of the most powerful tools for AI-SEO due to its data processing capabilities. You can clean your existing HTML content using libraries like markdownify and convert it into the format LLMs love most.

Python Automation Example

from flask import Flask, Response
from markdownify import markdownify as md

@app.route('/llms.txt')
def generate_llms_txt():
    raw_content = get_latest_articles() # Database function
    content = "# Articlesnn"
    for art in raw_content:
        markdown_body = md(art.html_content) # HTML to Markdown
        content += f"- [{art.title}]({art.url}): {markdown_body[:200]}...n"
    
    return Response(content, mimetype='text/plain')

Technical Checklist

• Content-Type Check: Ensure the file is served as text/plain.
• Robots.txt Integration: Increase discoverability by adding LLMS: /llms.txt below the Sitemap: line.
• Caching: Implement Redis or file-based 1-hour caching to avoid putting load on the database with every request.
• Hierarchy: The H1 heading should be your site name, and H2 headings should be the main categories.
• llms-full.txt: If your site is very large, create a second file for deep crawling and link to it from the main file.

Common Mistakes

• Leaving HTML Tags: AI agents waste tokens trying to interpret <div> or <script> tags. Use only Markdown.
• Broken Links: Links that are not updated when the URL structure changes in automatically generated files lower your AI credibility.
• Excessively Long Summaries: A page summary should not exceed 2-3 sentences. The goal is to tell the AI which page to read, not to copy the entire page.

In conclusion, the answer to how to write llms.txt lies not just in the file format, but in how 'fresh' that file is. By integrating one of the script methods above into your system, you can technically prepare your site for the AI revolution.