Wagtail Agent Ready

First-class markdown twins, authored /llms.txt and /robots.txt builder.

Opted-in Wagtail pages get a markdown representation you can template the same way as HTML. Pages without the mixin stay HTML-only.

This app offers an optional /llms.txt and /robots.txt editors in the admin. These apps have migrations, if you prefer you can handle them on your own.

What does it mean to be Agent Ready?

An agent-ready site is one AI agents can discover, parse, and interact with without a human browser. isitagentready.com scores that by probing emerging standards. Cloudflare's scanner groups them into five categories.

The scanner offers Content Site, API / Application, and All Checks presets.

Discoverability:

  • robots.txt (RFC 9309) that points at a sitemap;
  • sitemap.xml: RFC 8288
  • Link headers to machine-readable resources (api-catalog, service-desc, service-doc, describedby);
  • DNS for AI Discovery (DNS-AID)

Content accessibility:

  • Accept: text/markdown returns Content-Type: text/markdown; HTML stays the default.

Bot access control:

  • explicit AI crawler User-agent rules in robots.txt;
  • Content Signals (search, ai-input, ai-train);
  • Web Bot Auth, if the origin signs outbound bot requests.

Protocol discovery — API Catalog, OAuth/OIDC, OAuth Protected Resource, Auth.md, MCP Server Card, A2A Agent Card, Agent Skills, WebMCP, ARD.

Commerce — x402, MPP, UCP, ACP.

This package covers the content-site checks that belong in Wagtail:

  • markdown twins (.md URLs and Accept negotiation)
  • Link headers (rel="alternate" for twins, describedby whenever llms.txt is mounted) plus {% agent_ready_head %}
  • optional CMS editors for /llms.txt and /robots.txt (AI bot groups and Content Signals).

It does not ship a sitemap; Wagtail already has wagtail.contrib.sitemaps (see Sitemap) — nor /llms-full.txt, Web Bot Auth keys, protocol discovery, or commerce.
DNS-AID records are DNS work. Scan a Wagtail site with the Content Site preset.

After those pieces are live, scan the origin on isitagentready.com.

Features

  • Opt-in per page type via AgentReadyMixin
  • GET /{page-path}.md and Accept: text/markdown on the HTML URL
  • Canonical /index.md for the site root; /path/index.md 301s to /path.md
  • HTTP Header Link: rel="alternate" plus {% agent_ready_head %} helper tag
  • Pages use a twin .md template
  • Block rendering cascade: to_markdown() → twin .md template → StreamField walker
  • Link rewriting filter that points at twins only when the target is opted in (|to_markdown_url)
  • {% markdown_image %} twin of Wagtail {% image %}
  • Optional agent_ready.llms_txt: CMS editor and GET /llms.txt, including locale-prefixed copies
  • Optional agent_ready.robots_txt: CMS editor and origin-root GET /robots.txt

Requirements

  • Python ≥ 3.10
  • Django ≥ 5.2, < 6.1
  • Wagtail ≥ 6.3

Guide to make your site Agent Ready

Markdown twins and headers (Base installation)

pip install wagtail-agent-ready
  1. Add "agent_ready" to INSTALLED_APPS.
  2. Include the package URLs before wagtail_urls.
from agent_ready.urls import urlpatterns as agent_ready_urls

urlpatterns += i18n_patterns(
    # ...
    *agent_ready_urls, # <-- Add this just before wagtail_urls
    path("", include(wagtail_urls)),
    # ...
)
  1. Mix the mixin into page models that should have a twin:
from agent_ready.mixins import AgentReadyMixin


class HomePage(AgentReadyMixin, Page):
    pass
  1. In the HTML layout template load the agent ready templatetags with{% load agent_ready %}
  2. Add the {% agent_ready_head %} tag in the HTML layout template's <head>.
  3. Add a Markdown twin template next to each opted-in page’s HTML template:
myapp/templates/myapp/home_page.html  ->  myapp/templates/myapp/home_page.md

Block twins are optional. Without a .md next to the block’s HTML template, it will render like it would without an HTML template:

myapp/templates/stream_blocks/text_image.html  ->  myapp/templates/stream_blocks/text_image.md

Sitemap

Enable Wagtail's sitemap contribution (wagtail.contrib.sitemaps) or equivalent on the origin root so GET /sitemap.xml returns live HTML page locs:

from wagtail.contrib.sitemaps.views import sitemap

urlpatterns = [
    path("sitemap.xml", sitemap),
    # ...
]

robots.txt

There must be a valid robots.txt served on the origin root so GET /robots.txt returns the parsable instructions for crowlers.
Inside the robots.txt there should be a reference to the sitemap full path, and there should be AI Content Signals as well.

You can serve it as a static view or a txt django template.

from django.views.generic import TemplateView

urlpatterns = [
    path(
        "robots.txt",
        TemplateView.as_view(
            template_name="robots.txt",
            content_type="text/plain",
        ),
    ),
]

If you would like to use an UI builder, you can enable our robots.txt builder. To install it you need to:

  1. Add "agent_ready.robots_txt" to INSTALLED_APPS (in addition to "agent_ready").
  2. Mount the URLs on the origin root so GET /robots.txt is not locale-negotiated.
from agent_ready.robots_txt.urls import urlpatterns as robots_txt_urls

urlpatterns = [
    *robots_txt_urls,
    # ...
]

GET /robots.txt is chosen by host (Site.find_for_request). Edit the file under Settings → robots.txt. When more than one Wagtail Site exists, Settings lists the sites; a single-site install still opens the editor directly.

Each site has one document, checkout the editor to see what you can add. Installing the app is not enough, GET /robots.txt 404s until a document exists.

When GET /sitemap.xml is a mounted URL, the robots.txt automatically advertises it. A sitemap that exists only as /{lang}/sitemap.xml is not advertised.

llms.txt

While it's not directly required by the Cloudflare agent-ready tests, it is useful to fufill the describedby link requirement of the tests.

Opted-in HTML and markdown responses send rel="describedby" pointing at llms.txt when that URL is mounted. The covering path is /{lang}/llms.txt when the request is language-prefixed and that route exists; otherwise /llms.txt. A custom llms.txt view is advertised as soon as it resolves.

You can create and serve them using templates and views, by mounting them on the origin root, and on the i18n_patterns if your site is multilingual.

from django.views.generic import TemplateView

urlpatterns = [
    path(
        "llms.txt",
        TemplateView.as_view(
            template_name="llms.txt",
            content_type="text/plain",
        ),
    ),
]

We created an admin editor to author these files like they would be pages in the admin. To install it you must:

  1. Add "agent_ready.llms_txt" to INSTALLED_APPS (in addition to "agent_ready").
  2. Mount the URLs on the origin root so GET /llms.txt is not locale-negotiated:
from agent_ready.llms_txt.urls import urlpatterns as llms_txt_urls

urlpatterns = [
    *llms_txt_urls,
    # ...
]
  1. If the host uses i18n_patterns, include the same module inside those patterns as well, before the .md catch-all / wagtail_urls:
urlpatterns += i18n_patterns(
    *llms_txt_urls,
    # ...
)

A monolingual site only needs the root include.

GET /llms.txt is chosen by host (Site.find_for_request). Edit the file under Settings → llms.txt. When more than one Wagtail Site exists, Settings lists the sites; a single-site install still opens the editor directly. One document per site and locale. When the body is empty, Insert menu pages fills the editor from that site’s menu (root plus live pages shown in menus) for the document’s locale; Save is still required. Internal page links in the body become markdown twins when the page is opted in, and are emitted as absolute URLs. describedby is sent only when that host and locale would 200 — a title is present, and prefixed paths fall back to that site's default-locale document. Installing the app is not enough.

DNS-AID

Wagtail does not manage DNS records. Publish them at the DNS host following the DNS for AI Discovery (DNS-AID) specification.

Usage

Visit any opted-in live page with a .md suffix, or send Accept: text/markdown on the HTML URL.

Appending /index.md to a page URL redirects to the canonical twin. The homepage is /index.md (and /en/index.md when a locale prefix is in use).

In Markdown templates:

{% load agent_ready %}
{% markdown_frontmatter %}

# {{ page.title|collapse_whitespace|escape_markdown }}

{{ page.body_field|to_markdown }}
{% markdown_block page.body %}
[{{ other.title|collapse_whitespace|escape_markdown }}]({{ other|to_markdown_url }})
{% markdown_image photo fill-600x338 %}
{% markdown_image photo fill-600x338 alt=caption %}

{% markdown_frontmatter %} is opt-in. The default dict is title, search_description when filled, published (last_published_at, not first published), and locale when WAGTAIL_I18N_ENABLED. {% markdown_frontmatter custom %} dumps that mapping only (no merge). Override markdown_frontmatter() on a page type to add keys. Scalars and lists of scalars work without extras; nested mappings need pip install wagtail-agent-ready[yaml].

If a page has no twin, |to_markdown_url keeps the HTML URL.

{% markdown_image %} is the markdown twin of Wagtail {% image %}: a filter spec is required, alt= uses the same attribute slot, and HTML-only attrs (class, loading, as, …) are rejected. The walker still uses its own specs (ImageBlock fill-600x338, ImageChooser original, rich-text format filter_specs), not this tag.

Page twins emit site-relative links and image srcs. Set AGENT_READY_ABSOLUTE_MARKDOWN_URLS = True to absolutize both. /llms.txt always uses absolute URLs.

Set serve_html = False on a page type to 404 the HTML URL and still serve Markdown. Those responses are also noindex. .md URLs of pages that still serve HTML get the same header; Accept: text/markdown on the HTML URL does not, so a crawler cannot noindex the canonical page.

Custom choosers that should not implement to_markdown() can register a walker serializer from wagtail_hooks.py. Registration only affects the walker (to_markdown() and a twin .md template still win):

# myapp/wagtail_hooks.py
from agent_ready.markdown import register_markdown_serializer
from myapp.blocks import PersonChooserBlock


@register_markdown_serializer(PersonChooserBlock)
def serialize_person(block, value, context):
    if not value:
        return ""
    return value.name

Package layout

agent_ready/
  mixins.py                  AgentReadyMixin
  views.py                   serve_markdown
  urls.py                    .md urlpattern
  http.py                    Accept negotiation, Link / Vary helpers
  markdown/
    renderer.py              cascade: method > twin > walker
    serializers.py           render_basic → Markdown
    richtext.py              Draftail HTML → Markdown
    frontmatter.py           YAML fence (PyYAML optional)
    normalizer.py            whitespace
    urls.py                  gated twin URLs
  templatetags/agent_ready.py
  llms_txt/                  optional app: CMS editor + /llms.txt
  robots_txt/                optional app: CMS editor + /robots.txt

Development

Setup

# Clone the repo
git clone https://github.com/infofactory/wagtail-agent-ready.git
cd wagtail-agent-ready

# Install with uv
uv sync --extra testing --group test

Running tests

# Run tests
python testmanage.py test

# Run with coverage
python -m coverage run testmanage.py test
python -m coverage report -m

Running linting

ruff check .
ruff format --check .

Running tox (matrix testing)

tox

License

BSD-3-Clause. See LICENSE for details.

Project links

Data verified by PyPI on Sep 11, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for infofactory from gravatar.com infofactory

Credits

Author: infoFactory

GitHub Statistics

Data verified by PyPI on Sep 11, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

BSD-3-Clause
View SPDX License List

Requires

Python >=3.10

Provides Extra

testing yaml

Tags

agent cms django llms.txt markdown wagtail

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

wagtail_agent_ready-0.1.0.tar.gz (34.2 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

wagtail_agent_ready-0.1.0-py3-none-any.whl (45.6 kB view details)

Uploaded Python 3

File details

Details for the file wagtail_agent_ready-0.1.0.tar.gz.

File metadata

  • Download URL: wagtail_agent_ready-0.1.0.tar.gz
  • Upload date:
  • Size: 34.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for wagtail_agent_ready-0.1.0.tar.gz
Algorithm Hash digest
SHA256 22c5eb4c711a5c519e315285e50039e1b9887f7af03d24cc178e945494200473
MD5 6e27cfa74d4a799ffb7eb0d3d4b49799
BLAKE2b-256 904491addb4097572fa2779a4924f748a872e94448ae8c7bd47fa16670f11287

See more details on using hashes here.

Provenance

The following attestation bundles were made for wagtail_agent_ready-0.1.0.tar.gz:

Publisher: python-publish.yml on infofactory/wagtail-agent-ready

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file wagtail_agent_ready-0.1.0-py3-none-any.whl.

File metadata

File hashes

Hashes for wagtail_agent_ready-0.1.0-py3-none-any.whl
Algorithm Hash digest
SHA256 cc803645ccfdbc74de0e80518911c10e4d72d86f6a42d53cbf9c2812502ff174
MD5 874b5ab8fdc203fef9cbcfd8bcd7e354
BLAKE2b-256 bf8ee2f281c89a81c1c2f25768242a048652bc7a831ebdb95f411fcd6feb9579

See more details on using hashes here.

Provenance

The following attestation bundles were made for wagtail_agent_ready-0.1.0-py3-none-any.whl:

Publisher: python-publish.yml on infofactory/wagtail-agent-ready

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page