canonicalwebteam.store-llm

Flask extension making a Canonical store (snapcraft.io, charmhub.io, ...) readable by LLMs:

• every public page is served as Markdown at /page.md
• /llms.txt lists the public pages, discovered from the routing table (llmstxt.org)
• /llms-full.txt concatenates their Markdown
• <link rel="alternate" type="text/markdown"> can be advertised from the layout

The HTML to Markdown conversion is done by canonicalwebteam.markdown-response; this package adds the store-specific wiring around it.

Installation

pip install canonicalwebteam.store-llm

Usage

from canonicalwebteam.store_llm import StoreLLM

StoreLLM(
    app,
    base_url="https://snapcraft.io",
    site_name="Snapcraft",
    summary="Snapcraft is the home of the Snap Store, where ...",
    sections={
        "": "Main pages",
        "about": "Publishing a snap",
        "docs": "Documentation",
        "store": "Store",
    },
    section_order=["Main pages", "Publishing a snap", "Documentation", "Store"],
    extra_links=[
        {
            "section": "Documentation",
            "url": "https://snapcraft.io/docs/",
            "title": "Snap documentation",
            "description": "Reference and explanation for snaps and snapd.",
        },
        {
            "section": "Optional",
            "url": "https://snapcraft.io/llms-full.txt",
            "title": "Every page in one file",
            "description": "The pages above concatenated as Markdown.",
        },
    ],
)

The application factory pattern works too: store_llm = StoreLLM(base_url=..., ...) then store_llm.init_app(app).

Options

What the store needs to provide

Templates. Discovery parses templates rather than rendering them, so every public page must define meta_title and meta_description blocks with static text. Variables render as nothing and an {% if %} contributes its {% else %} branch, so a template shared between a listing and a detail page can keep a static default:

{% block meta_title %}About Snaps | Snapcraft{% endblock %}
{% block meta_description %}Snaps are app packages for desktop, cloud and IoT.{% endblock %}

{# listed as "Interface catalogue" #}
{% block meta_title %}{% if interface %}{{ interface.name }} interface{% else %}Interface catalogue{% endif %}{% endblock %}

A page opts out of llms.txt with <meta name="robots" content="noindex">, the same signal that keeps it out of search results. Layouts and partials (file names starting with _ or containing layout) are never listed, and a meta_title a layout defines is only a fallback: it does not list pages that have no title of their own.

Content. Markdown is extracted from <main id="main-content"> (falling back to <body>). Add data-md-strip to interactive chrome that reads badly as text: copy buttons, share widgets, charts.

Alternate link. Templates get markdown_path, the Markdown URL path of the current page or None for private pages:

{% if markdown_path %}
  <link rel="alternate" type="text/markdown" href="https://snapcraft.io{{ markdown_path }}" />
  <link rel="describedby" href="https://snapcraft.io/llms.txt" />
{% endif %}

Sitemap. app.extensions["store_llm"].sitemap_paths() returns the same pages, so the sitemap and llms.txt never drift.

Build step

Rendering every page for llms-full.txt is too slow to do on request, so generate both files at build time and the routes serve them from the static folder (falling back to rendering live when the files are missing, for development):

python3 -m canonicalwebteam.store_llm generate webapp.app:create_app

The argument is module:attribute, either a Flask app or a factory returning one. Run it from the repository root after the frontend build, for example in the Dockerfile and rockcraft.yaml, and ignore static/llms.txt and static/llms-full.txt in git. The command reports pages without a meta_description and pages that could not be rendered.

Customising the output

The files are rendered from the store_llm/llms.txt and store_llm/llms-full.txt templates. A store can override either by adding a template with the same name to its own templates/ folder; both receive site_name, and llms.txt receives summary and sections ([{"section", "links": [{"title", "url", "description"}]}]).

How pages are discovered

A route is listed when it takes no URL parameters, answers GET, is not private, its view calls render_template("....html") with a literal name and does not redirect, and the template (or a layout it extends) has a non-empty meta_title. Routes rendering the same template or sharing a title are listed once, under the shortest path (/store, not /explore). The homepage is left out: the header of llms.txt already introduces the site.