Skip to main content

django-robots-manager

Tests GitHub tag Python Django License Ruff pre-commit

A continuation of django-robots, picking up from the 6.x series (based on release 6.1), after a long pause in releases of the original library.

The import path and app label stay robots, so it is a drop-in replacement for django-robots 5.0 and 6.x: existing tables and migration history are reused.

Changes from upstream 6.1

  • __version__ is read via importlib.metadata only; the pkg_resources fallback and default_app_config (Django < 3.2) are removed.
  • The South guard in robots.migrations is removed.
  • RuleAdminForm rejects a rule whose allowed and disallowed URLs share a pattern.
  • RuleAdminForm rejects a second rule for the same robot (case-insensitive) on the same site.
  • ROBOTS_USE_HOST now defaults to False: Yandex stopped using the Host directive in 2018, and Google never supported it (it is not part of RFC 9309 either). Set it to True to keep the old output.
  • ROBOTS_SITE_BY_REQUEST looks the site up the same way as Django's sites framework: case-insensitively, retrying without the port (example.com:8000 matches example.com).
  • robots.txt is served as text/plain; charset=utf-8 (RFC 9309 requires UTF-8), with rules sorted by robot and URLs by pattern, in a fixed number of queries.
  • Rule.comment: an optional single-line note rendered as a # ... line above the rule's group in robots.txt.
  • URL patterns are percent-encoded on output: non-ASCII characters (e.g. Cyrillic), whitespace and # become %XX, as Yandex requires. Patterns are stored as entered, and raw and encoded forms of the same path count as the same pattern when checking allowed/disallowed conflicts.
  • Clean-param directives (Yandex): see below.
  • System checks for ROBOTS_SITEMAP_URLS: robots.E001 if it is a string instead of a list, robots.W001 for URLs that are not absolute, robots.W002 for non-ASCII domains.
  • Only the ru locale is shipped.

Installation

Requires Python 3.10+ and Django 4.2+ (tested with Django 4.2, 5.2, 6.0 and 6.1). Projects on older Python or Django versions can stay on django-robots 6.1.

INSTALLED_APPS = [
    "django.contrib.sites",
    ...
    "robots",
]
urlpatterns = [
    re_path(r"^robots\.txt", include("robots.urls")),
]

Settings (ROBOTS_SITEMAP_URLS, ROBOTS_USE_SITEMAP, ROBOTS_USE_HOST, ROBOTS_CACHE_TIMEOUT, ROBOTS_SITE_BY_REQUEST, ROBOTS_USE_SCHEME_IN_HOST, ROBOTS_SITEMAP_VIEW_NAME) are the same as upstream, except for the changes listed above.

Internationalized domains

Store non-ASCII domains in Site.domain (and in ROBOTS_SITEMAP_URLS) in Punycode, e.g. xn--d1aqf.xn--p1ai instead of дом.рф:

  • the domain is written to Sitemap: (and Host:) as stored, and Yandex requires Punycode there;
  • with ROBOTS_SITE_BY_REQUEST = True the site is looked up by the request Host header, which is always Punycode, so a site stored as дом.рф is not found and robots.txt responds with an error.

Clean-param

Clean-param tells Yandex which URL parameters do not change the page content, so /catalog/?ref=vk and /catalog/?sid=1 are crawled and indexed as /catalog/. Other search engines ignore it.

Add directives in the admin under Clean-param directives and attach them to sites:

Parameters Path Output
ref&sid /catalog/ Clean-param: ref&sid /catalog/
sort (empty) Clean-param: sort (whole site)

The directive is cross-sectional, so it is rendered once per file, next to Sitemap, not inside a User-agent group. Following the Yandex rules, parameter names are case-sensitive, the path may contain only A-Za-z0-9.-/*_ (a leading / is added if missing, as for URL patterns), and the whole line is limited to 500 characters. Unlike Allow/Disallow, the path is not percent-encoded, so pages with Cyrillic paths (e.g. /о-компании/) can only be covered by a directive without a path.

Development

tests/ holds a minimal Django project (SQLite) used both for the test suite and for trying the app locally. It is not part of the distributed package. Run the commands from the repository root.

python -m venv .venv
. .venv/bin/activate
pip install -e .
export DJANGO_SETTINGS_MODULE=tests.settings

python -m django test tests          # run the test suite

pip install "coverage[toml]"         # test coverage, as in CI
python -m coverage run -m django test tests
python -m coverage report

python -m django migrate
python -m django createsuperuser
python -m django runserver           # http://localhost:8000/admin/, http://localhost:8000/robots.txt

python -m django makemigrations robots --check --dry-run   # after model changes

The demo project uses SITE_ID = 1 (example.com), so rules must be attached to that site to appear in /robots.txt.

Linting and formatting use ruff via pre-commit; neither is a dependency of the package.

pip install pre-commit
pre-commit install                   # run the hooks on every commit
pre-commit run --all-files           # run them on the whole repository

Metadata

Release files for django-robots-manager 6.2.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for django-robots-manager 6.2.1
File Size Uploaded Provenance Actions
django_robots_manager-6.2.1.tar.gz 18.4 kB Without provenance Details for django_robots_manager-6.2.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for django-robots-manager 6.2.1
File Interpreter ABI Platform Actions
django_robots_manager-6.2.1-py3-none-any.whl Python 3 none any Details for django_robots_manager-6.2.1-py3-none-any.whl

Total release size: 43.5 kB

Release files / django_robots_manager-6.2.1.tar.gz

Download URL django_robots_manager-6.2.1.tar.gz
Size 18.4 kB
Tags Source
SHA-256 checksum
How to use checksums
0efff9f4d038b8612c32ce331953e2f59b342becad819cf57617a2ed254cdb8e
BLAKE2b-256 checksum
How to use checksums
98dcedf88b059dbef2ca38904f66550ace6b8253e74e7a1fa1b917b1627a57fe
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/2.4.1 CPython/3.12.3 Linux/6.8.0-138-generic

Release files / django_robots_manager-6.2.1-py3-none-any.whl

Download URL django_robots_manager-6.2.1-py3-none-any.whl
Size 25.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
8eaf58e08e2076fd891710079d1f3fbb4a7b8997e33c0bd38caf94251c387ca4
BLAKE2b-256 checksum
How to use checksums
4008d10beb301de33bc0577113d31b50afceb74a5b517e4702fb9052bedae011
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/2.4.1 CPython/3.12.3 Linux/6.8.0-138-generic

Security for django-robots-manager 6.2.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

6.2.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page