wagtail-machine-readable 0.3.0

Make your Wagtail site AI-ready. llms.txt, Markdown page variants and structured data in one install.

wagtail-machine-readable

Make your Wagtail site AI-ready. llms.txt, Markdown page variants and structured data in one install.

LLMs, answer engines and AI crawlers are becoming the front door to your content — and they read llms.txt, not your carefully tuned templates. wagtail-machine-readable turns your existing Wagtail page tree into spec-compliant machine-readable outputs automatically: live pages only, privacy restrictions respected, every Site in a multi-site install served on its own hostname. Install it, add two lines, and /llms.txt works.

What you get

• /llms.txt — a spec-compliant index of your site: H1 site name, blockquote description, H2 sections derived from your top-level pages, with - [name](url): description link lists.
• /llms-full.txt — the same page set with page content rendered to Markdown.
• .md page variants — every visible page served as Markdown on its own URL (/about/team.md, /index.md for the site root).
• StreamField→Markdown extraction — headings, links, images (alt text
  • URL), embeds and tables survive extraction instead of being stripped.
• JSON-LD structured data — schema.org WebPage/Article, Organization and BreadcrumbList per page via a template tag, with a per-page-type builder mapping.
• AI crawler robots.txt — opt-in allow/deny controls for known AI user agents (GPTBot, ClaudeBot, PerplexityBot, …) managed from settings.
• Static export — a management command that writes every output to disk for static hosting and CDN workflows.
• Wagtail-native visibility rules — only live pages, view-restricted (private) subtrees excluded, drafts excluded, multi-site aware, plus per-model and per-page opt-outs.

Quickstart (60 seconds)

pip install wagtail-machine-readable

# settings.py
INSTALLED_APPS = [
    # …
    "wagtail_machine_readable",
]

# urls.py — before Wagtail's catch-all page serving include
urlpatterns = [
    # …
    path("", include("wagtail_machine_readable.urls")),
    path("", include(wagtail_urls)),
]

That's it. Zero configuration produces a valid llms.txt:

# Acme

> Acme makes modular widgets.

## Pages

- [Contact](https://example.com/contact/): Get in touch.

## About

- [About](https://example.com/about/): Who we are.
- [Team](https://example.com/about/team/): The people.

## Blog

- [Blog](https://example.com/blog/): News and articles.
- [First post](https://example.com/blog/first-post/): Our first post.
- [Second post](https://example.com/blog/second-post/)

Sections come from the children of each Site's root page; descriptions come from search_description by default. Every Wagtail Site gets its own document on its own hostname.

Markdown page variants

Every visible page is also served as Markdown by appending .md to its slug path — /about/team/ becomes /about/team.md, and the site root is /index.md. Responses use text/markdown and the same visibility rules as llms.txt, so drafts and private pages 404. Set MARKDOWN_ENABLED to False to turn the variants off.

Bodies are produced by MarkdownContentExtractor, which maps rich text and StreamField blocks to Markdown — headings (demoted below the page title), [text](url) links with rich-text page references expanded, ![alt](url) images, embeds and URL blocks as autolinks, and TableBlock values as Markdown tables.

Structured data (JSON-LD)

Add the template tag to your page template:

{% load machine_readable %}
{% structured_data page %}

This renders a <script type="application/ld+json"> element containing a schema.org @graph: the page node (WebPage by default), a BreadcrumbList from the page's ancestors and an Organization derived from the Wagtail Site. Map page types to other builders — Article ships ready to use — or your own StructuredDataBuilder subclasses:

WAGTAIL_MACHINE_READABLE = {
    "STRUCTURED_DATA_BUILDERS": {
        "blog.BlogPage": "wagtail_machine_readable.structured_data.ArticleBuilder",
    },
}

Mappings match base classes too, so "wagtailcore.Page" changes the default for every page type.

AI crawler robots.txt

Opt in to serving /robots.txt with explicit rules for known AI user agents:

WAGTAIL_MACHINE_READABLE = {
    "ROBOTS_TXT_ENABLED": True,
    "ROBOTS_AI_DEFAULT": "allow",      # baseline for known AI agents
    "ROBOTS_AI_DENY": ["Bytespider"],  # per-agent overrides
    "ROBOTS_EXTRA": "Sitemap: https://example.com/sitemap.xml",
}

The built-in list covers GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot, Meta-ExternalAgent and friends; the allow/deny lists also accept agents not on the list. Other crawlers get User-agent: * / Allow: /. If your project already serves robots.txt, keep its URL pattern above the package include.

Static generation

python manage.py generate_machine_readable --output ./static-export
python manage.py generate_machine_readable --site example.com

Single-site projects write llms.txt, llms-full.txt, the .md page tree (index.md, about/team.md, …) and — when enabled — robots.txt into the output directory; multi-site projects get one subdirectory per hostname.

Caching

Responses carry Cache-Control: public, max-age=3600 by default (CACHE_MAX_AGE). For sites where generation itself is expensive, opt in to caching the generated documents server-side:

WAGTAIL_MACHINE_READABLE = {
    "GENERATION_CACHE_TIMEOUT": 86400,
}

Rendered documents are stored in Django's default cache and invalidated when a page is published or unpublished, so a long timeout is safe.

Settings

All configuration lives in a single dict. Every key has a working default — configure only what you want to change:

WAGTAIL_MACHINE_READABLE = {
    "SITE_DESCRIPTION": "Acme makes modular widgets.",
    "MAX_PAGES_PER_SECTION": 25,
    "EXCLUDE_PAGE_MODELS": ["blog.BlogTagIndexPage"],
}

Key	Default	Purpose
SITE_DESCRIPTION	None	Blockquote description. A string, or a {hostname: description} dict for multi-site. Falls back to the root page's description chain.
SECTION_STRATEGY	TopLevelSectionStrategy path	Dotted path to the class deriving H2 sections from the page tree.
MAX_PAGES_PER_SECTION	50	Cap entries per section (None = unlimited).
RESPECT_SHOW_IN_MENUS	False	Only include pages with show_in_menus=True.
EXCLUDE_PAGE_MODELS	[]	Page models to exclude, as "app_label.ModelName" strings (exact type).
EXCLUDE_PAGE_IDS	[]	Specific page ids to exclude.
DESCRIPTION_FIELDS	["machine_readable_description", "search_description"]	Per-page description fallback chain; first non-empty field wins.
FULL_TEXT_ENABLED	True	Serve/write llms-full.txt.
FULL_TEXT_MAX_PAGES	None	Cap the number of pages in llms-full.txt (None = unlimited).
MARKDOWN_ENABLED	True	Serve/write .md page variants.
EXTRACTOR	MarkdownContentExtractor path	Dotted path to the ContentExtractor used for page bodies.
CACHE_MAX_AGE	3600	Cache-Control: public, max-age=N on responses (0 = no header).
GENERATION_CACHE_TIMEOUT	None	Opt-in server-side caching of generated documents, in seconds.
STRUCTURED_DATA_BUILDERS	{}	Map "app_label.ModelName" to StructuredDataBuilder dotted paths.
ROBOTS_TXT_ENABLED	False	Serve/write robots.txt from this package.
ROBOTS_AI_DEFAULT	"allow"	Baseline policy ("allow"/"deny") for known AI user agents.
ROBOTS_AI_ALLOW	[]	Agents to explicitly allow, overriding the baseline.
ROBOTS_AI_DENY	[]	Agents to explicitly deny, overriding the baseline.
ROBOTS_EXTRA	""	Raw text appended to robots.txt (e.g. a Sitemap: line).

Misconfigured keys are caught by Django system checks at startup.

Customisation

Exclude a page type — no mixin or migration needed:

class InternalToolPage(Page):
    exclude_from_machine_readable = True

Per-page editor controls — adopt the optional mixin for a dedicated AI-consumer description and a per-page exclusion flag, surfaced in the page editor by MachineReadablePanel:

from wagtail_machine_readable.models import MachineReadableMixin
from wagtail_machine_readable.panels import MachineReadablePanel

class ArticlePage(MachineReadableMixin, Page):
    promote_panels = Page.promote_panels + [MachineReadablePanel()]

The mixin adds fields to your page model, so run python manage.py makemigrations && python manage.py migrate after adopting it (and after upgrading from a version with fewer mixin fields). Until the migration is applied, any query against the page model — including the pages themselves — fails with a missing-column error.

Custom sections — subclass SectionStrategy and point SECTION_STRATEGY at it:

from wagtail_machine_readable.generators import SectionStrategy

class NavigationSections(SectionStrategy):
    def build_sections(self, site):
        ...

Custom content extraction — subclass ContentExtractor (or MarkdownContentExtractor, whose block handling and convert_html hook are overridable) and point EXTRACTOR at it. Set EXTRACTOR to "wagtail_machine_readable.extractors.DefaultContentExtractor" for the plain-text behaviour of v0.1.

Custom structured data — subclass StructuredDataBuilder (or WebPageBuilder/ArticleBuilder) and map page types to it via STRUCTURED_DATA_BUILDERS.

How it behaves

• Visibility: a page appears only if it is live, has no view restriction on itself or an ancestor, and is not excluded by settings, the class attribute or the per-page flag. The private/draft rules match what anonymous visitors can already see — nothing non-public leaks.
• Multi-site: Site.find_for_request() scopes every request, so each hostname serves its own tree with absolute URLs.
• Caching: responses carry Cache-Control: public, max-age=3600 by default. The views are cache_page-compatible (cache keys include the host), so wrapping them or enabling Django's cache middleware works per site out of the box. Server-side generation caching is opt-in via GENERATION_CACHE_TIMEOUT, invalidated on publish/unpublish.
• Output safety: titles and descriptions are collapsed to single lines and escaped, so page content cannot inject sections or links into the document structure. JSON-LD payloads escape <, > and &, so page content cannot break out of the script element.

How is this different from django-llms-txt?

django-llms-txt is Django-generic: you describe your content to it. wagtail-machine-readable is Wagtail-native: it already understands the page tree (sections for free), live/privacy rules, multi-site scoping, search_description, and StreamField content. Point it at a Wagtail project and it produces the right documents with zero configuration.

Roadmap

v0.2 (Markdown page variants) and v0.3 (structured data and crawler controls) have shipped. See ROADMAP.md for what's next.

Compatibility

Python 3.11–3.13 · Django 4.2/5.2 · Wagtail 6.3–7.x. The full matrix is tested in CI.

Contributing

git clone https://github.com/brett-allard-amp/wagtail-machine-readable.git
cd wagtail-machine-readable
python -m venv .venv && source .venv/bin/activate
pip install -e . --group dev
pre-commit install
pytest

Run the full matrix with tox. Bug reports, spec-compliance issues and extractor improvements are all welcome — please include a failing test where possible.

License

MIT