Skip to main content

django-robots-manager

Tests PyPI Downloads Python Django License Ruff pre-commit

A continuation of django-robots, picking up from the 6.x series (based on release 6.1), after a long pause in releases of the original library.

The import path and app label stay robots, so it is a drop-in replacement for django-robots 5.0 and 6.x: existing tables and migration history are reused.

Changes from upstream 6.1

  • __version__ is read via importlib.metadata only; the pkg_resources fallback and default_app_config (Django < 3.2) are removed.
  • The South guard in robots.migrations is removed.
  • RuleAdminForm rejects a rule whose allowed and disallowed URLs share a pattern.
  • RuleAdminForm rejects a second rule for the same robot (case-insensitive) on the same site.
  • ROBOTS_USE_HOST now defaults to False: Yandex stopped using the Host directive in 2018, and Google never supported it (it is not part of RFC 9309 either). Set it to True to keep the old output.
  • ROBOTS_SITE_BY_REQUEST looks the site up the same way as Django's sites framework: case-insensitively, retrying without the port (example.com:8000 matches example.com).
  • robots.txt is served as text/plain; charset=utf-8 (RFC 9309 requires UTF-8), with rules sorted by robot and URLs by pattern, in a fixed number of queries.
  • Rule.comment: an optional single-line note rendered as a # ... line above the rule's group in robots.txt.
  • URL patterns are percent-encoded on output: non-ASCII characters (e.g. Cyrillic), whitespace and # become %XX, as Yandex requires. Patterns are stored as entered, and raw and encoded forms of the same path count as the same pattern when checking allowed/disallowed conflicts.
  • Clean-param directives (Yandex): see below.
  • System checks for ROBOTS_SITEMAP_URLS: robots.E001 if it is a string instead of a list, robots.W001 for URLs that are not absolute, robots.W002 for non-ASCII domains.
  • Only the ru locale is shipped.

Installation

Requires Python 3.10+ and Django 4.2+. CI is configured to check all 22 officially supported combinations of the declared Python and Django series, using the latest available patch release of each Django series. The pairs follow Django's compatibility table.

Python Django series
3.10 4.2, 5.0, 5.1, 5.2
3.11 4.2, 5.0, 5.1, 5.2
3.12 4.2, 5.0, 5.1, 5.2, 6.0, 6.1
3.13 5.1, 5.2, 6.0, 6.1
3.14 5.2, 6.0, 6.1
3.15 6.1

Projects on older Python or Django versions can stay on django-robots 6.1.

INSTALLED_APPS = [
    "django.contrib.sites",
    ...
    "robots",
]
urlpatterns = [
    re_path(r"^robots\.txt", include("robots.urls")),
]

Settings (ROBOTS_SITEMAP_URLS, ROBOTS_USE_SITEMAP, ROBOTS_USE_HOST, ROBOTS_CACHE_TIMEOUT, ROBOTS_SITE_BY_REQUEST, ROBOTS_USE_SCHEME_IN_HOST, ROBOTS_SITEMAP_VIEW_NAME) are the same as upstream, except for the changes listed above.

Internationalized domains

Store non-ASCII domains in Site.domain (and in ROBOTS_SITEMAP_URLS) in Punycode, e.g. xn--d1aqf.xn--p1ai instead of дом.рф:

  • the domain is written to Sitemap: (and Host:) as stored, and Yandex requires Punycode there;
  • with ROBOTS_SITE_BY_REQUEST = True the site is looked up by the request Host header, which is always Punycode, so a site stored as дом.рф is not found and robots.txt responds with an error.

Clean-param

Clean-param tells Yandex which URL parameters do not change the page content, so /catalog/?ref=vk and /catalog/?sid=1 are crawled and indexed as /catalog/. Other search engines ignore it.

Add directives in the admin under Clean-param directives and attach them to sites:

Parameters Path Output
ref&sid /catalog/ Clean-param: ref&sid /catalog/
sort (empty) Clean-param: sort (whole site)

The directive is cross-sectional, so it is rendered once per file, next to Sitemap, not inside a User-agent group. Following the Yandex rules, parameter names are case-sensitive, the path may contain only A-Za-z0-9.-/*_ (a leading / is added if missing, as for URL patterns), and the whole line is limited to 500 characters. Unlike Allow/Disallow, the path is not percent-encoded, so pages with Cyrillic paths (e.g. /о-компании/) can only be covered by a directive without a path.

Development

tests/ holds a minimal Django project (SQLite) used both for the test suite and for trying the app locally. It is not part of the distributed package. Run the commands from the repository root.

python -m venv .venv
. .venv/bin/activate
pip install -e .
export DJANGO_SETTINGS_MODULE=tests.settings

python -m django test tests          # run the test suite

pip install "coverage[toml]"         # test coverage, as in CI
python -m coverage run -m django test tests
python -m coverage report

python -m django migrate
python -m django createsuperuser
python -m django runserver           # http://localhost:8000/admin/, http://localhost:8000/robots.txt

python -m django makemigrations robots --check --dry-run   # after model changes

The demo project uses SITE_ID = 1 (example.com), so rules must be attached to that site to appear in /robots.txt.

Linting and formatting use ruff via pre-commit; neither is a dependency of the package.

pip install pre-commit
pre-commit install                   # run the hooks on every commit
pre-commit run --all-files           # run them on the whole repository

Releasing

  1. Bump version in pyproject.toml and commit.
  2. Tag the commit as v<version> and push the tag.

The Release workflow checks that the tag matches the version, builds the package and publishes it to TestPyPI and then to PyPI via trusted publishing.

Metadata

Release files for django-robots-manager 6.2.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for django-robots-manager 6.2.2
File Size Uploaded Provenance Actions
django_robots_manager-6.2.2.tar.gz 19.0 kB With provenance Details for django_robots_manager-6.2.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for django-robots-manager 6.2.2
File Interpreter ABI Platform Actions
django_robots_manager-6.2.2-py3-none-any.whl Python 3 none any Details for django_robots_manager-6.2.2-py3-none-any.whl

Total release size: 44.5 kB

Release files / django_robots_manager-6.2.2.tar.gz

Download URL django_robots_manager-6.2.2.tar.gz
Size 19.0 kB
Tags Source
SHA-256 checksum
How to use checksums
cf49a1b4141f58a73e16f40577164138a97d7bad52ac4e9761831000e457d1fe
BLAKE2b-256 checksum
How to use checksums
5f9cfaa176949d1d004d43225ab2e315ff4f516ebe1cd006b0c070221a386ceb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.

Transparency log

Release files / django_robots_manager-6.2.2-py3-none-any.whl

Download URL django_robots_manager-6.2.2-py3-none-any.whl
Size 25.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
15c671c0054e082e2e4e6a1f82d178bfd797eaf1dda519feb8dc99ab555e6fe5
BLAKE2b-256 checksum
How to use checksums
87c04898bec36e995aaca3875e5478b6433e92166a7ec89d89f9b740933187af
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.

Transparency log

Security for django-robots-manager 6.2.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for django_robots_manager-6.2.2.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

6.2.2 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page