wagtail-machine-readable 0.1.0

Make your Wagtail site AI-ready. llms.txt, Markdown page variants and structured data in one install.

wagtail-machine-readable

Make your Wagtail site AI-ready. llms.txt, Markdown page variants and structured data in one install.

LLMs, answer engines and AI crawlers are becoming the front door to your content — and they read llms.txt, not your carefully tuned templates. wagtail-machine-readable turns your existing Wagtail page tree into spec-compliant machine-readable outputs automatically: live pages only, privacy restrictions respected, every Site in a multi-site install served on its own hostname. Install it, add two lines, and /llms.txt works.

What you get

• /llms.txt — a spec-compliant index of your site: H1 site name, blockquote description, H2 sections derived from your top-level pages, with - [name](url): description link lists.
• /llms-full.txt — the same page set with page content extracted to plain text/Markdown.
• Static export — a management command that writes the same files to disk for static hosting and CDN workflows.
• Wagtail-native visibility rules — only live pages, view-restricted (private) subtrees excluded, drafts excluded, multi-site aware.

Quickstart (60 seconds)

pip install wagtail-machine-readable

# settings.py
INSTALLED_APPS = [
    # …
    "wagtail_machine_readable",
]

# urls.py — before Wagtail's catch-all page serving include
urlpatterns = [
    # …
    path("", include("wagtail_machine_readable.urls")),
    path("", include(wagtail_urls)),
]

That's it. Zero configuration produces a valid llms.txt:

# Acme

> Acme makes modular widgets.

## Pages

- [Contact](https://example.com/contact/): Get in touch.

## About

- [About](https://example.com/about/): Who we are.
- [Team](https://example.com/about/team/): The people.

## Blog

- [Blog](https://example.com/blog/): News and articles.
- [First post](https://example.com/blog/first-post/): Our first post.
- [Second post](https://example.com/blog/second-post/)

Sections come from the children of each Site's root page; descriptions come from search_description by default. Every Wagtail Site gets its own document on its own hostname.

Static generation

python manage.py generate_machine_readable --output ./static-export
python manage.py generate_machine_readable --site example.com

Single-site projects write llms.txt/llms-full.txt into the output directory; multi-site projects get one subdirectory per hostname.

Settings

All configuration lives in a single dict. Every key has a working default — configure only what you want to change:

WAGTAIL_MACHINE_READABLE = {
    "SITE_DESCRIPTION": "Acme makes modular widgets.",
    "MAX_PAGES_PER_SECTION": 25,
    "EXCLUDE_PAGE_MODELS": ["blog.BlogTagIndexPage"],
}

Key	Default	Purpose
SITE_DESCRIPTION	None	Blockquote description. A string, or a {hostname: description} dict for multi-site. Falls back to the root page's description chain.
SECTION_STRATEGY	TopLevelSectionStrategy path	Dotted path to the class deriving H2 sections from the page tree.
MAX_PAGES_PER_SECTION	50	Cap entries per section (None = unlimited).
RESPECT_SHOW_IN_MENUS	False	Only include pages with show_in_menus=True.
EXCLUDE_PAGE_MODELS	[]	Page models to exclude, as "app_label.ModelName" strings (exact type).
EXCLUDE_PAGE_IDS	[]	Specific page ids to exclude.
DESCRIPTION_FIELDS	["machine_readable_description", "search_description"]	Per-page description fallback chain; first non-empty field wins.
FULL_TEXT_ENABLED	True	Serve/write llms-full.txt.
EXTRACTOR	DefaultContentExtractor path	Dotted path to the ContentExtractor used for full-text bodies.
CACHE_MAX_AGE	3600	Cache-Control: public, max-age=N on responses (0 = no header).

Misconfigured keys are caught by Django system checks at startup.

Customisation

Exclude a page type — no mixin or migration needed:

class InternalToolPage(Page):
    exclude_from_machine_readable = True

Dedicated descriptions for AI consumers — adopt the optional mixin:

from wagtail_machine_readable.models import MachineReadableMixin

class ArticlePage(MachineReadableMixin, Page):
    promote_panels = Page.promote_panels + [
        FieldPanel("machine_readable_description"),
    ]

Custom sections — subclass SectionStrategy and point SECTION_STRATEGY at it:

from wagtail_machine_readable.generators import SectionStrategy

class NavigationSections(SectionStrategy):
    def build_sections(self, site):
        ...

Custom content extraction — subclass ContentExtractor and point EXTRACTOR at it. The v0.2 StreamField→Markdown renderer will ship behind this same interface.

How it behaves

• Visibility: a page appears only if it is live, has no view restriction on itself or an ancestor, and is not excluded by settings or the class attribute. The private/draft rules match what anonymous visitors can already see — nothing non-public leaks.
• Multi-site: Site.find_for_request() scopes every request, so each hostname serves its own tree with absolute URLs.
• Caching: responses carry Cache-Control: public, max-age=3600 by default. The views are cache_page-compatible (cache keys include the host), so wrapping them or enabling Django's cache middleware works per site out of the box.
• Output safety: titles and descriptions are collapsed to single lines and escaped, so page content cannot inject sections or links into the document structure.

How is this different from django-llms-txt?

django-llms-txt is Django-generic: you describe your content to it. wagtail-machine-readable is Wagtail-native: it already understands the page tree (sections for free), live/privacy rules, multi-site scoping, search_description, and — on the roadmap — StreamField content. Point it at a Wagtail project and it produces the right document with zero configuration.

Roadmap

• v0.2 — .md variants of every page and a full StreamField→Markdown renderer.
• v0.3 — JSON-LD structured data and AI crawler robots.txt controls.

See ROADMAP.md for details.

Compatibility

Python 3.11–3.13 · Django 4.2/5.2 · Wagtail 6.3–7.x. The full matrix is tested in CI.

Contributing

git clone https://github.com/brett-allard-amp/wagtail-machine-readable.git
cd wagtail-machine-readable
python -m venv .venv && source .venv/bin/activate
pip install -e . --group dev
pre-commit install
pytest

Run the full matrix with tox. Bug reports, spec-compliance issues and extractor improvements are all welcome — please include a failing test where possible.

License

MIT