django-robots-manager
A continuation of django-robots, picking up from the 6.x series (based on release 6.1), after a long pause in releases of the original library.
The import path and app label stay robots, so it is a drop-in replacement for
django-robots 5.0 and 6.x: existing tables and migration history are reused.
Changes from upstream 6.1
__version__is read viaimportlib.metadataonly; thepkg_resourcesfallback anddefault_app_config(Django < 3.2) are removed.- The South guard in
robots.migrationsis removed. RuleAdminFormrejects a rule whose allowed and disallowed URLs share a pattern.RuleAdminFormrejects a second rule for the same robot (case-insensitive) on the same site.ROBOTS_USE_HOSTnow defaults toFalse: Yandex stopped using theHostdirective in 2018, and Google never supported it (it is not part of RFC 9309 either). Set it toTrueto keep the old output.ROBOTS_SITE_BY_REQUESTlooks the site up the same way as Django's sites framework: case-insensitively, retrying without the port (example.com:8000matchesexample.com).robots.txtis served astext/plain; charset=utf-8(RFC 9309 requires UTF-8), with rules sorted by robot and URLs by pattern, in a fixed number of queries.Rule.comment: an optional single-line note rendered as a# ...line above the rule's group inrobots.txt.- URL patterns are percent-encoded on output: non-ASCII characters (e.g. Cyrillic),
whitespace and
#become%XX, as Yandex requires. Patterns are stored as entered, and raw and encoded forms of the same path count as the same pattern when checking allowed/disallowed conflicts. - Clean-param directives (Yandex): see below.
- System checks for
ROBOTS_SITEMAP_URLS:robots.E001if it is a string instead of a list,robots.W001for URLs that are not absolute,robots.W002for non-ASCII domains. - Only the
rulocale is shipped.
Installation
Requires Python 3.10+ and Django 4.2+ (tested with Django 4.2, 5.2, 6.0 and 6.1). Projects on older Python or Django versions can stay on django-robots 6.1.
INSTALLED_APPS = [
"django.contrib.sites",
...
"robots",
]
urlpatterns = [
re_path(r"^robots\.txt", include("robots.urls")),
]
Settings (ROBOTS_SITEMAP_URLS, ROBOTS_USE_SITEMAP, ROBOTS_USE_HOST,
ROBOTS_CACHE_TIMEOUT, ROBOTS_SITE_BY_REQUEST, ROBOTS_USE_SCHEME_IN_HOST,
ROBOTS_SITEMAP_VIEW_NAME) are the same as upstream, except for the changes listed
above.
Internationalized domains
Store non-ASCII domains in Site.domain (and in ROBOTS_SITEMAP_URLS) in Punycode,
e.g. xn--d1aqf.xn--p1ai instead of дом.рф:
- the domain is written to
Sitemap:(andHost:) as stored, and Yandex requires Punycode there; - with
ROBOTS_SITE_BY_REQUEST = Truethe site is looked up by the requestHostheader, which is always Punycode, so a site stored asдом.рфis not found androbots.txtresponds with an error.
Clean-param
Clean-param tells Yandex which URL parameters do not change the page content, so
/catalog/?ref=vk and /catalog/?sid=1 are crawled and indexed as /catalog/. Other
search engines ignore it.
Add directives in the admin under Clean-param directives and attach them to sites:
| Parameters | Path | Output |
|---|---|---|
ref&sid |
/catalog/ |
Clean-param: ref&sid /catalog/ |
sort |
(empty) | Clean-param: sort (whole site) |
The directive is cross-sectional, so it is rendered once per file, next to Sitemap,
not inside a User-agent group. Following the Yandex rules, parameter names are
case-sensitive, the path may contain only A-Za-z0-9.-/*_ (a leading / is added
if missing, as for URL patterns), and the whole line is limited to 500 characters.
Unlike Allow/Disallow, the path is not percent-encoded, so pages with Cyrillic
paths (e.g. /о-компании/) can only be covered by a directive without a path.
Development
tests/ holds a minimal Django project (SQLite) used both for the test suite and for
trying the app locally. It is not part of the distributed package. Run the commands
from the repository root.
python -m venv .venv
. .venv/bin/activate
pip install -e .
export DJANGO_SETTINGS_MODULE=tests.settings
python -m django test tests # run the test suite
pip install "coverage[toml]" # test coverage, as in CI
python -m coverage run -m django test tests
python -m coverage report
python -m django migrate
python -m django createsuperuser
python -m django runserver # http://localhost:8000/admin/, http://localhost:8000/robots.txt
python -m django makemigrations robots --check --dry-run # after model changes
The demo project uses SITE_ID = 1 (example.com), so rules must be attached to that
site to appear in /robots.txt.
Linting and formatting use ruff via pre-commit; neither is a dependency of the package.
pip install pre-commit
pre-commit install # run the hooks on every commit
pre-commit run --all-files # run them on the whole repository
Metadata
Release files for django-robots-manager 6.2.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| django_robots_manager-6.2.1.tar.gz | 18.4 kB | Without provenance | Details for django_robots_manager-6.2.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| django_robots_manager-6.2.1-py3-none-any.whl | Python 3 | none | any | Details for django_robots_manager-6.2.1-py3-none-any.whl |
Total release size: 43.5 kB
Release files / django_robots_manager-6.2.1.tar.gz
| Download URL | django_robots_manager-6.2.1.tar.gz |
|---|---|
| Size | 18.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0efff9f4d038b8612c32ce331953e2f59b342becad819cf57617a2ed254cdb8e
|
|
BLAKE2b-256 checksum How to use checksums |
98dcedf88b059dbef2ca38904f66550ace6b8253e74e7a1fa1b917b1627a57fe
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.4.1 CPython/3.12.3 Linux/6.8.0-138-generic
|
Release files / django_robots_manager-6.2.1-py3-none-any.whl
| Download URL | django_robots_manager-6.2.1-py3-none-any.whl |
|---|---|
| Size | 25.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
8eaf58e08e2076fd891710079d1f3fbb4a7b8997e33c0bd38caf94251c387ca4
|
|
BLAKE2b-256 checksum How to use checksums |
4008d10beb301de33bc0577113d31b50afceb74a5b517e4702fb9052bedae011
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.4.1 CPython/3.12.3 Linux/6.8.0-138-generic
|
Security for django-robots-manager 6.2.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.