wagtail-machine-readable

Make your Wagtail site AI-ready. llms.txt, Markdown page variants and structured data in one install.

LLMs, answer engines and AI crawlers are becoming the front door to your content — and they read llms.txt, not your carefully tuned templates. wagtail-machine-readable turns your existing Wagtail page tree into spec-compliant machine-readable outputs automatically: live pages only, privacy restrictions respected, every Site in a multi-site install served on its own hostname. Install it, add two lines, and /llms.txt works.

What you get

• /llms.txt — a spec-compliant index of your site: H1 site name, blockquote description, H2 sections derived from your top-level pages, with - [name](url): description link lists.
• /llms-full.txt — the same page set with page content rendered to Markdown.
• .md page variants — every visible page served as Markdown on its own URL (/about/team.md, /index.md for the site root), with content negotiation and rel="alternate" links so agents can find them.
• StreamField→Markdown extraction — headings, links, images (alt text
  • URL), embeds and tables survive extraction instead of being stripped.
• JSON-LD structured data — schema.org WebPage/Article, Organization and BreadcrumbList per page via a template tag, with a per-page-type builder mapping.
• AI crawler robots.txt — opt-in allow/deny controls for known AI user agents (GPTBot, ClaudeBot, PerplexityBot, …) managed from settings.
• AI crawler analytics — see which AI bots read your content in an admin report, recorded by an opt-in middleware.
• Editor tools — a "Machine readable content" admin report and a page-editor preview mode showing pages as machines see them.
• Static export — a management command that writes every output to disk for static hosting and CDN workflows.
• Wagtail-native visibility rules — only live pages, view-restricted (private) subtrees excluded, drafts excluded, multi-site aware, plus per-model and per-page opt-outs.

Quickstart (60 seconds)

pip install wagtail-machine-readable

# settings.py
INSTALLED_APPS = [
    # …
    "wagtail_machine_readable",
]

# urls.py — before Wagtail's catch-all page serving include
urlpatterns = [
    # …
    path("", include("wagtail_machine_readable.urls")),
    path("", include(wagtail_urls)),
]

That's it. Zero configuration produces a valid llms.txt:

# Acme

> Acme makes modular widgets.

## Pages

- [Contact](https://example.com/contact/): Get in touch.

## About

- [About](https://example.com/about/): Who we are.
- [Team](https://example.com/about/team/): The people.

## Blog

- [Blog](https://example.com/blog/): News and articles.
- [First post](https://example.com/blog/first-post/): Our first post.
- [Second post](https://example.com/blog/second-post/)

Sections come from the children of each Site's root page; descriptions come from search_description by default. Every Wagtail Site gets its own document on its own hostname.

Markdown page variants

Every visible page is also served as Markdown by appending .md to its slug path — /about/team/ becomes /about/team.md, and the site root is /index.md. Responses use text/markdown and the same visibility rules as llms.txt, so drafts and private pages 404. Set MARKDOWN_ENABLED to False to turn the variants off.

Bodies are produced by MarkdownContentExtractor, which maps rich text and StreamField blocks to Markdown — headings (demoted below the page title), [text](url) links with rich-text page references expanded, ![alt](url) images, embeds and URL blocks as autolinks, and TableBlock values as Markdown tables.

Making the variants discoverable

Crawlers won't guess your URL convention, so advertise the variants. Add the middleware:

MIDDLEWARE = [
    # …
    "wagtail_machine_readable.middleware.MachineReadableMiddleware",
]

This does two things for canonical page URLs:

• Content negotiation — a request with Accept: text/markdown ranked above HTML gets the Markdown variant directly (with Vary: Accept and a Content-Location header). Browsers never send this, so regular visitors are unaffected.
• Link headers — HTML page responses gain Link: <…/team.md>; rel="alternate"; type="text/markdown".

And in your page templates, advertise the variant in the head:

{% load machine_readable %}
{% markdown_alternate page %}

which renders <link rel="alternate" type="text/markdown" href="…">. The middleware resolves the page for each negotiated or HTML page response, costing one extra query — skip it (and keep the template tag) if that matters at your scale.

Structured data (JSON-LD)

Add the template tag to your page template:

{% load machine_readable %}
{% structured_data page %}

This renders a <script type="application/ld+json"> element containing a schema.org @graph: the page node (WebPage by default), a BreadcrumbList from the page's ancestors and an Organization derived from the Wagtail Site. Map page types to other builders — Article ships ready to use — or your own StructuredDataBuilder subclasses:

WAGTAIL_MACHINE_READABLE = {
    "STRUCTURED_DATA_BUILDERS": {
        "blog.BlogPage": "wagtail_machine_readable.structured_data.ArticleBuilder",
    },
}

Mappings match base classes too, so "wagtailcore.Page" changes the default for every page type.

AI crawler robots.txt

Opt in to serving /robots.txt with explicit rules for known AI user agents:

WAGTAIL_MACHINE_READABLE = {
    "ROBOTS_TXT_ENABLED": True,
    "ROBOTS_AI_DEFAULT": "allow",      # baseline for known AI agents
    "ROBOTS_AI_DENY": ["Bytespider"],  # per-agent overrides
    "ROBOTS_EXTRA": "Sitemap: https://example.com/sitemap.xml",
}

The built-in list covers GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot, Meta-ExternalAgent and friends; the allow/deny lists also accept agents not on the list. Other crawlers get User-agent: * / Allow: /. If your project already serves robots.txt, keep its URL pattern above the package include.

Editor tools

With wagtail.admin installed the package adds two reports (under Reports in the admin menu):

• Machine readable content — every live page with its machine-readable status: description present, extracted body non-empty, excluded from outputs. Editors work through the "missing" and "empty" rows.
• AI crawler activity — recent requests from known AI crawlers (see below).

For a "view as machine" preview inside the page editor, adopt the preview mixin (plain class, no migration):

from wagtail_machine_readable.models import MachineReadablePreviewMixin

class ArticlePage(MachineReadablePreviewMixin, Page):
    ...

Editors get a "Machine readable" entry in the preview-mode dropdown showing the page exactly as its .md variant renders — garbled extraction becomes visible while editing, not after publishing.

AI crawler analytics

Prove the machine-readable outputs are being read. Add the tracking middleware:

MIDDLEWARE = [
    # …
    "wagtail_machine_readable.middleware.AICrawlerTrackingMiddleware",
]

Requests whose User-Agent matches a known AI crawler (the same list the robots.txt controls use, plus any agents in your allow/deny settings) are recorded with agent, host, path and status code, and surfaced in the AI crawler activity report with CSV/XLSX export. The hit model ships with its own migration, so run python manage.py migrate after installing or upgrading. Keep the table tidy from a cron job:

python manage.py prune_ai_crawler_hits --days 90

Static generation

python manage.py generate_machine_readable --output ./static-export
python manage.py generate_machine_readable --site example.com

Single-site projects write llms.txt, llms-full.txt, the .md page tree (index.md, about/team.md, …) and — when enabled — robots.txt into the output directory; multi-site projects get one subdirectory per hostname.

Caching

Responses carry Cache-Control: public, max-age=3600 by default (CACHE_MAX_AGE). For sites where generation itself is expensive, opt in to caching the generated documents server-side:

WAGTAIL_MACHINE_READABLE = {
    "GENERATION_CACHE_TIMEOUT": 86400,
}

Rendered documents are stored in Django's default cache and invalidated when a page is published or unpublished, so a long timeout is safe.

Settings

All configuration lives in a single dict. Every key has a working default — configure only what you want to change:

WAGTAIL_MACHINE_READABLE = {
    "SITE_DESCRIPTION": "Acme makes modular widgets.",
    "MAX_PAGES_PER_SECTION": 25,
    "EXCLUDE_PAGE_MODELS": ["blog.BlogTagIndexPage"],
}

Misconfigured keys are caught by Django system checks at startup.

Customisation

Exclude a page type — no mixin or migration needed:

class InternalToolPage(Page):
    exclude_from_machine_readable = True

Per-page editor controls — adopt the optional mixin for a dedicated AI-consumer description and a per-page exclusion flag, surfaced in the page editor by MachineReadablePanel:

from wagtail_machine_readable.models import MachineReadableMixin
from wagtail_machine_readable.panels import MachineReadablePanel

class ArticlePage(MachineReadableMixin, Page):
    promote_panels = Page.promote_panels + [MachineReadablePanel()]

The mixin adds fields to your page model, so run python manage.py makemigrations && python manage.py migrate after adopting it (and after upgrading from a version with fewer mixin fields). Until the migration is applied, any query against the page model — including the pages themselves — fails with a missing-column error.

Custom sections — subclass SectionStrategy and point SECTION_STRATEGY at it:

from wagtail_machine_readable.generators import SectionStrategy

class NavigationSections(SectionStrategy):
    def build_sections(self, site):
        ...

Custom content extraction — subclass ContentExtractor (or MarkdownContentExtractor, whose block handling and convert_html hook are overridable) and point EXTRACTOR at it. Set EXTRACTOR to "wagtail_machine_readable.extractors.DefaultContentExtractor" for the plain-text behaviour of v0.1.

Custom structured data — subclass StructuredDataBuilder (or WebPageBuilder/ArticleBuilder) and map page types to it via STRUCTURED_DATA_BUILDERS.

How it behaves

• Visibility: a page appears only if it is live, has no view restriction on itself or an ancestor, and is not excluded by settings, the class attribute or the per-page flag. The private/draft rules match what anonymous visitors can already see — nothing non-public leaks.
• Multi-site: Site.find_for_request() scopes every request, so each hostname serves its own tree with absolute URLs.
• Caching: responses carry Cache-Control: public, max-age=3600 by default. The views are cache_page-compatible (cache keys include the host), so wrapping them or enabling Django's cache middleware works per site out of the box. Server-side generation caching is opt-in via GENERATION_CACHE_TIMEOUT, invalidated on publish/unpublish.
• Output safety: titles and descriptions are collapsed to single lines and escaped, so page content cannot inject sections or links into the document structure. JSON-LD payloads escape <, > and &, so page content cannot break out of the script element.

How is this different from django-llms-txt?

django-llms-txt is Django-generic: you describe your content to it. wagtail-machine-readable is Wagtail-native: it already understands the page tree (sections for free), live/privacy rules, multi-site scoping, search_description, and StreamField content. Point it at a Wagtail project and it produces the right documents with zero configuration.

Roadmap

v0.2 (Markdown page variants) and v0.3 (structured data and crawler controls) have shipped. See ROADMAP.md for what's next.

Compatibility

Python 3.11–3.13 · Django 4.2/5.2 · Wagtail 6.3–7.x. The full matrix is tested in CI.

Contributing

git clone https://github.com/brett-allard-amp/wagtail-machine-readable.git
cd wagtail-machine-readable
python -m venv .venv && source .venv/bin/activate
pip install -e . --group dev
pre-commit install
pytest

Run the full matrix with tox. Bug reports, spec-compliance issues and extractor improvements are all welcome — please include a failing test where possible.

License

MIT